DOCUMENT RESUHE 



ED 270 589 



CE 044 527 



AUTHOR 
TITLE 

INSTITUTION 

SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

AVAILABLE FROM 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Dubravcic, Elizabeth V.; And Others 
Assessing Vocational Teachers. Research and 
Development Series No. 262. 

Ohio State Univ., Columbus. National Center for 

Research in Vocational Education. 

Office of Vocational and Adult Education (ED), 

Washington, DC. 

86 

300-83-0016 
95p. 

National Center 
for Research in 
Road, Colximbus, 



Publications, Box F, National Center 
Vocational Education, 1960 Kenny 
OH 43210-1090 (RD262— $8 . 00 ) . 



Guides - Non-Classroom Use (055) — Viewpoints (120) 
MF01/PC04 Plus Postage. 

Academic Achievement; Classroom Observation 
Techniques; Comparative Analysis; Competence; 
Educational Policy; *Evaluation Criteria; ^Evaluation 
Methods; Interviews; Peer Evaluation; Policy 
Formation; Postsecondary Education; Secondary 
Education; Self Evaluation (Individuals); State 
Action; ^Statewide Planning; Student Evaluation of 
Teacher Performance; Teacher Attitudes;, *Teacher 
Evaluation; Teacher Improvement; Teacher Recruitment; 
Teacher Selection; *Vocational Education; ^Vocational 
Education Teachers 



ABSTRACT 

This combination report and guide is intended to 
assist a broad audience of state and local educational 
administrators, teacher educators, and state policymakers. The first 
chapter of the guide examines (1) current views from the field 
regarding procedures for defining teacher effectiveness, teacher 
supply and demand, factors affecting teacher quantity and quality, 
and the changing demands on vocational education; and (2) strategies 
for improving teacher recruitment, selection, and certification. 
Provided in the next chapter are action agendas for state 
legislatures, state departments of education, and teacher preparation 
institutions to implement in their efforts to improve the 
effectiveness of vocational teachers and teaching. The third chapter 
of the guide describes and assesses the following teacher evaluation 
strategies: teacher competency testing, teacher interviews, student 
achievement, classroom observation, student rating of teachers, peer 
review, and self-evaluation. Appendixes to the report include lists 
of technical advisory panel members and site visit locations. 
References are provided at the conclusion of chapter 1 and following 
each of the evaluation sections of chapter 3. (MN) 

*************************************************** ******^ 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
*********************************************************************** 



ERIC 



Research and Development Series No. 262 



00 

in 
o 

rsJ 
o 

UJ 



ASSESSING VOCATIONAL TEACHERS 



Elizabeth V. Dubravcic 
Christian A. Chinien 
Frank C. Pratzner 



0 



ERIC 



The National Center for Research in Vocational Education 
The Ohio State University 

1960 Kenny Road 
Columbus, Ohio 43210-1090 



1986 



2 



n« "*?^'**»^*«ENT OF EDUCATION 
Offy of Educaf.onai Rese«fCh and Improvement 

EDjdCATIONAL RESOURCES INFORMATION 
y CENTER (ERIC) 

CrThis clocument has been reproduced a$ 
received from the person or OfOtnization 
originating it 

O Minor char^oes have been made to improve 
reproductior) quality 



Points of view or opinions slated ir, thisdocu 
meni do -^ot necessarily represent ofdctal 
OERl position or policy 



THE NATIONAL CENTER MISSION STATEMENT 



The National Center for Research in Vocational Education's mission is to increase the ability of 
diverse agencies, institutions, and organizations to solve educational problems relating to 
individual career planning, preparation, and progression. The National Center fulflils its n^iission 
by: 

• Generating knowledge through research 

• Developing educational programs and products 

• Evaluating individual program needs and outcomes 

• i^roviding information for national planning and policy 

• Installing educational programs and products 

• Operating ir formation systems and services 

o Conducting leadership development and training programs 



For further information contact: 
Program Information Office 

National Center for Research in Vocational Education 
The Ohio State University 
1960 Kenny Road 
Columbus. Ohio 43210 

Telephone: (614) 486-3655 or (800) 848-4815 
Cable: CTVOCEDOSU/Columbus. Ohio 
Telex: 8104821894 



ERIC 



3 



FUNDING INFORMATION 



Proiect rnit: 

Contrac Number 

Pro|e€t I lumber 

Act und ir Which 
Funds A dmlnlstered: 



The National Center for Research in Vocational Education, 
Evaluation 

300830016 

051OC50010 

Education Amendments of 1976, P.L. 94-482 



Source of Contract: 



Contractjor 



Executive Director 



Disclaimer 



Discrimination 
Prohlbited:| 



Office of Vocational and Adult Education 
U.S. Department of Education 
Washington, D.C. 20202 

The National Center for Research 

in Vocational Education 
The Ohio State University 
Coiumbus, Ohio 43210*1090 

Robert E. Taylor 

This publication was prepared pursuant to a contract with the Office 
of Vocational and Adult Education, U.S. Department of Education. 
Contractors undertaking such projects under Government sponsor- 
ship are encouraged to express freely their judgment in professional 
and technical matters. Points of view or opinions do not, therefore, 
necessarilv represent official U.S. Department of Education position 
or policy. 

Title VI of the Civil Rights Act of 1964 states: "No person in the 
United States shall, on the ground of race, color, or national origin, 
be excluded from participation in, be denied the benefits of, or be 
subjected to discrimination under any program or activity receiving 
Federal financial assistance." Title IX of the Education Amendments 
of 1972 states: "No person in the United States shall, on the basis of 
sex, be excluded from participation in, be denied the benefits of, or 
oe subjected to discrimination under any education program or 
activity receiving Federal financial assistance." Therefore, the 
National Center for Research in Vocational Education Project. Nice 
every program or project funded by the U.S. Department of Educa* 
tion, must be operated in compliance with these laws. 



ERIC 



4 



TABLE OF CONTENTS 



LIST OF FIGURES ^ 

FOREWORD 

EXECUTIVE SUMMARY 

CHAPTER 1. INTRODUCTION ^ 

Overview of the Publication 2 

Audience ^ 

Statement of the Problem ^ 

Current Views from the Field ^ 

Defining Teacher Effectiveness ^ 

Teacher Supply and Demand 5 

Factors Affecting Teacher Quantity and Quality 6 

Changing Demands on Vocational Education ^ 

Improvement Strategies ' ^ 

Recruitment and Selection ^ 

Teacher Certification ^ 

References 

CHAPTER 2. AN ACTION AGENDA FOR IMPROVING THE EFFECTIVENESS OF 

VOCATIONAL TEACHERS AND TEACHING 13 

State Legislatures ^3 

State Departments of Education and Vocational Education 13 

Teacher Shortages and Surpluses • 1^ 

Certification and Recertification 1^ 

Teacher Preparation Programs 15 

School-Based Improvement 15 

InternaK i^ff Development 16 

Teacher Preparation Institutions 16 

Recruitment. Selection, and Screening of Students 16 

Programs and Curriculum 1^ 

Quality of Faculty 

Evaluation of Programs 1^ 

Outreach Activities 20 



ERLC 



CHAPTER 3. TEACHER EVALUATION STRATEGIES 21 

The Need to Focus Teacher Evaluation 21 

Teacher Competency Testing 24 

Competency Testing for Selective Admission 24 

Competency Testing for Certification 27 

Conclusion 32 

References 3^ 

Teacher Interviews 3^ 

Teacher Selection Interviews 37 

Teacher Appraisal Interviews 38 

References ^0 

Student Achievement 41 

Overview ^'^ 

Issues in Use 42 

Advantages and Disadvantages 45 

Conditions for Use in Vocational Education 45 

References 49 

Classroom Observation 50 

Introduction 50 

Classroom Visits 50 

Systematic Observation 52 

References 63 

Student Rating Teachers 64 

Description 64 

Approaches and Techniques 64 

Issues in Use 65 

Advantages and Disadvantages 67 

Applications in Vocational Education - 68 

References ^3 

Peer Review ^4 

Description ^4 

Issues in Use 

Advantages and Disadvantages 78 

Applications in Vocational Education 79 

References 80 

Self-Evaluation 

Description ^'^ 

Issues in Use - 82 

Advantages and Disadvantages 82 

Applications in Vocational Education 83 

References 84 

APPENDIX A. TECHNICAL ADVISORY PANEL MEMBERS 85 

APPENDIX B. SITE VISIT LOCATIONS 87 



iv 



LIST OF FIGURES 

1. The Teacher Occupational Competency Test (TOCT) 33 

2. The National Evaluation Systems' Vocational Teacher Competency Testing 34 

3. The Student Occupational Competency Achievement Test (SOCAT) 47 

4. The Ohio Vocational Education Achievement Test Program (OVEAT) 48 

5. The Classroom Observation Keyed for Effective Research (COKER) 58 

6. The Observation Schedule and Record, Form 5 (0SCAR5V) 59 

7. Planning/Observation instrument (P/Ol) 60 

8. The Student Time-on-Task Observation Guides 61 

9. The Flanders System of Interaction Analysis 62 

10. The Student Perceptions Instrument 69 

11. The Student Perceptions of Teacher Style Instrument (SPOTS) 70 

12. The Classroom Climate Questionnaire (CCQ) 71 

13. Student Instruction Attitude Inventory 72 



V 

7 



FOREWORD 



Jmproving tOw-HCher effectiveness has been targeted for major reforms in recent United States 
education policy. Virtually every State has adopted, or is in the process of adopting, new programs 
and policies improving the recruitment, selection, preparation, certification, inservice develop- 
ment, and evaluation of its teaching work force. 

This publication examines the implications these reform initiatives have for vocational 
teachers and teaching. It provides decision makers at the State and local levels, as well as voca- 
tional teacher educators, with an examination of critical issues related to improving policy and 
practice in areas directly affecting vocational teachers. 

We wish to thank the Office of Vocational and Adult Education, U.S. Department of Education, 
for sponsoring the research project that produced this document. The project was conducted in 
the Evaluation and Policy Division of the National Center for Research in Vocational Education 
under the leadership of N. L. McCaslin, Associate Director. We wish to thank the project staff— 
Frank C. Pratzner, Project Director; Elizabeth V. Dubravcic, Program Associate; and Christian 
Chinien, Graduate Research Associate— for their work in preparing the document. 

We are especially grateful to Hazel Crain, University of Nebraska; Glen Fardig, University of 
Central Florida; Lonnie McNatt, Arkansas State Department of Vocational Education; and Thomas 
Walker, Temple University; for serving as members of the project's technical panel. Their advice 
and recommendations were most helpful, and many have been incorporated directly in the text. 

We also extend our appreciation to William Lehrer, Superintendent, and Kenneth Searfoss, 
Executive Director, Vocational, Technical and Career Education, Toledo Public Schools; Louis 
Woods, Principal, and Kay Ladd, Assistant Principal, Macomber Vocational High School of the 
Toledo Public Schools; Dal Lawrence of the Toledo Federation of Teachers, Toledo, Ohio; Kay 
Mitchell, Director of the Career Development Program, and Betty Standish, Director of Vocational 
Education of Charlotte-Mecklenburg School System, Charlotte, North Carolina; and Bill Warner of 
District 916 Area Vocational-Technical Institute, White Bear Lake, Minnesota. These individuals 
and their staffs coordinated and hosted visits and provided invaluable information at interviews. 

We were fortunate to have benefited from the critical reviews and recommendations of Bob 
Soar, University of Florida, and Thomas Walker, Temple University, external reviewers, and from 
the reviews of National Center staff members Dewey Adams and James Weber. Although we do 
not mention them by name, we also thank the many individuals in State departments of education 
and vocational education, educational researchers, and National Center staff who supplied us with 
papers or other printed documentation relating to vocational teacher development, certification, 
and evaluation. 

Finally, this document benefited from the long hours of typing by Jeannette L. Painter, Cathy 
Jones, Lou Pierson. and Colleen Kinzelman. We are grateful for their help and for the editing of 
Janet Kiplinger and her staff. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 
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EXECUTIVE SUMMARY 



One of the major focal points of the recent school reform movement has been improving the 
effectiveness of teachers and teaching. Many of the proposed strategies— improving recruitment 
and selection, improving preparation, improving certification and recertification, induction into 
teaching, and developing career ladders and merit pay— imply an understanding of what consti- 
tutes teacher effectiveness and a valid and reliable mechanism for its assessment At the present 
time.we have only a partial understanding of what constitutes effectiveness in vocational teaching. 

Improving and maintaining vocational teacher effectiveness requires a broad-based strategy 
aimed at assuring teacher competence and performance; assuring a sufficient supply of competent 
vocational teachers to meet the demands: and providing teachers with adequate resources and 
flexible support to enable them to be most effective in promoting student learning. State legisla- 
tures. State departments of education, State departments of vocational education, and teacher 
preparation must work together and with teachers and local administrators to help define goals 
and roles and develop appropriate strategies. It is important to note that any real and substantive 
changes or improvements can come only through a joint, collaborative effort involving school- 
based representatives* as well as State and regional officials snd teacher educators. 

Teacher evaluation is a cornerstone for many strategies aimed at improving teacher compe- 
tence and performance. Seven broad approaches to teacher evaluation include teacher compe- 
tency testing, teacher interviews, student achievement, classroom observation, student r9tings. 
peer review, and self-evaluation. Teacher competency testing may be used for purposes of screen- 
ing and selection to teacher preparation, leacher certification, and promotion. Tests provide a 
standard' zed, quick method of assessing a wide range of cognitive knowledge and some compe- 
tency. Their drawbacks lie in their validity for assessing teaching effectiveness and in the ques- 
tions of equity in their use for screening. Teacher interviews are most frequently used in selection 
and appraisals of teachers. They can provide an excellent mechanism for two-way information 
exchange that permits a quick focusing on areas of special interest or concern. Student achieve- 
ment provides the ultimate test of teacher effectiveness. Yet, using this appioach to evaluate 
teachers is plagued with many problems of measurement. 

Classroom observation is the most commonly used method for assessing teachers. While 
single class visits can uncover gross teacher incompetencies, systematic observation using expe- 
rienced teachers trained in observation provides one of the most promising approaches to improv- 
ing effectiveness. Student ratings can provide useful feedback on the effects of teaching, as well as 
a powerful incentive for teachers to change or develop. In using student ratings as an evaluative 
tool, care must be taken to differentiate between the effects of affective impressions and 
skill/cognitive development and between contextual factors within the classroom and those factors 
directly attributable to teaching. 

Peer review is gaining recognition as a valid approach to evaluating teachers at the elementary 
and secondary levels. The key to its success lies in selecting and training experts in evaluation and 
in providing sufficient resources (teacher time) for the process. This method is particu^ariy useful 
for improving performance of novice teachers. 

Self-evaluation can be the most powerful tool for formative development provided the teacher 
has the resources, skills, motivation, and support to obtain and accurately interpret feedback 
about his/her own teaching performance. 

Taken individually, each of these strategies has its advantages, disadvantages, and conditions 
or constraints for use. Taken collectively and applied with appropriate caution, they provide voca- 
tional administrators and policymakers with substantial options in meeting their evaluation needs. 
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CHAPTER 1 
INTRODUCTION 



In recent years a great deal of public attention has focused upon the decline in the quality of 
the American education system. Reports of declining student achievement scores on standardized 
tests (particularly in mpth, science, and reading)have alerted the public to problems facing ele- 
mentary and secondary schools. Numerous National and State commissions and task forces have 
been charged to investigate these problems and to make recommendations about their solution. 
Griesemer and Butler (1983) have summarized nine of the major National reports and have found 
their concerns and recommendations focusing on eight major areas: school organization and 
management, curriculum, student attitudes and special learning needs, quality and equality, 
teachers and teaching, postsecondary education, leadership, and research. 

In vocational education, the report of the National Commission on Secondary Vocational Edu- 
cation entitled The Unfinished Agenda. The Role of Vocational Education in the High School 
(1984) reviewed the problems of secondary vocational education and identified the following areas 
needing reform: 

• Perceptions of vocational education 

• Access 

• Curriculum 

• Teacher education and recruitment 

• Standards and accountability 

• Articulation 

• Leadership 

• Business, labor, and community involvement 

• Field-based learning, including cooperative education 

Although some of these areas of concern are unique to vocational education, it is clear that voca- 
tional education needs to address many of the same concerns as education in general. 

Of all of these areas, it is the effectiveness of teach()rs and teaching that has caused the great- 
est public concern and been the major focus of educational reforms in recent years. Kaplan (1985) 
observes that teachers and teaching have been "portrayed as constituting a near-disaster area in 
education*'(p. 2). He observes further that 

the recruitment, performance, work habits, incentives, preparation, and quality of 
teachers have ignited attention and action throughout the nation. Scarcely a week 
passes without legislative or executive measures aimed at achieving excellence in the 
teaching profession . . . 700 pieces of state legislation in 1983 and 1984. (p. 2) 
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States have tried to improve teacher quality by requiring tests, more stringent certification and 
recertification, and reassessment of teacher preparation programs, as well as by trying to recruit 
more academically able students into teaching. School districts have begun to move in the direc- 
tion of linking salaries to evaluation in some form of merit pay or a career ladder system. 

It has become clear that many of the strategies initially proposed were geared toward provid- 
ing the public with an expedient ''measure of accountability" rather than with the knowledge base 
necessary to identify and implement effective strategies for improving teachers and teaching. 
Furthermore, many of these strategies presuppose the existence of valid, reliable forms of teacher 
assessment, an assumption that both the research literature and current practice indicate is with- 
out basis. 

This publication reviews both the broad question of improving teacher effectiveness, as well as 
one of Its key components, the evaluation of teachers and teaching. It explores both areas with the 
primary intent of improving vocational teachers and teaching. To date, very little has been written 
that directly addresses the improvement needs of vocational teachers, and literature on assess- 
ment of vocational teachers is scarce. 

The study that resulted in this publication examined these two major areas in the following 
manner. First, a review of current literature was undertaken to identify current reforms in improv- 
ing teaching effectiveness and, in particular, teacher assessment practices. Second, a panel of 
experts was convened for 2 days at the National Center for Research in Vocational Education to 
examine various reforms and to recommend ways in which teacher preparation institutions and 
State departments of vocational education could assist in improving the effectiveness of vocational 
teaching. The panel was composed of representatives from both State departments of vocational 
education and teacher preparation institutions (appendix A lists the panel members). Third, five 
sites were visited by project staff to examine and document exemplary practices in teacher evalua- 
tion and improvement (sites are identified in appendix B). 



Overview of the Publication 

This publication is a synthesis of the information collected through the review of the literature, 
panel recommendations, and field site observations. The first chapter looks at the various prob- 
lems and issues pertaining to vocational teachers and the quality of their teaching. It examines 
some of the proposed strategies for improving teacher effectiveness and reviews some improve- 
ment strategies used in the field. 

Chapter 2 addresses specific actors in the improvement process: State departments of educa- 
tion and vocational education. State legislatures, and teacher preparation institutions and local 
education agencies at the secondary and postsecondary levels. It proposes a broad agenda for 
action to be considered by each of these actors to improve vocational teacher effectiveness. 

Chapter 3 examines the state of vocational teacher evaluation. It discusses seven approaches 
to evaluation, their advantages and disadvantages, and conditions for their appropriate use. 



Audience 

The first section of the publication is designed primarily for a broad audience of State and 
local educational administrators, teacher educators, and State policymakers. The issues raised in 
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the first chapter, along with the action agenda proposed in the second, emphasize the importance 
of collaboratior. ;n designing, implementing, and evaluating teacher improvement progr^Tis if any 
real long-term solutions are to ensue. 

The evaluation approaches in chapter 3 were initially examined to help local-level administra- 
tors select from among the multitude of possible evaluation techniques available and to determine 
which teacher evaluation systems are most appropriate to their school's goals and needs. During 
the course of the study, however, the researchers found that many drawbacks exist in current eval- 
uation approaches and that many gaps exist in the current knowledge base. Therefore, although 
the local administrators will still find this chapter of use. the third chapter is addressed also to 
educational researchers and policymakers. 



Statement oC the Problem 

The quality of teaching is seen by many as the Key to improvement in the schools. The 
National Commission on Excellence in Education (1983) says in its findings on teaching that 

not enough of the academicalty able students are being attracted to teaching; that 
teacher preparation programs need substantial improvement: that the professional work- 
ing life of teachers is on the whole unacceptable and thai a serious shortage of teachers 
exists in key fields, (p. 22) 

The commission also noted these specific problems associated with the teaching profession: 

• Too many teachers are being drawn from the bottom quarter of graduating high school 
and college classes. 

• The teacher preparation curriculum is weighted heavily with courses in "educational 
nfiethods*' at the expense of courses in subjects to be taught. A survey of 1.350 institutions 
training teachers indicated that 41 percent of the time of elementary school teacher can- 
didates is spent in education courses, which reduces the amount of time available for 
subjdct-matter courses. 

• The average salary after 12 years cf teaching is only $17,000 per year, and many teachers 
are required to supplement their income with part-time and summer employment. In addi- 
tion, individual teachers have tittle influence in such critical professional decisions as. for 
example, textbook selection. 

• Despite widespread publicity about an overpopulation of teachers, severe shortages of 
certain kinds of teachers exist: in the fields of mathematics, science, and foreign lan- 
guages and among specialists in education for gifted and talented, language minority, 
and handicapped students. 

• The siiortage of teachers in mathematics andscience is particularly severe A 1981 survey 
of 45 States revealed shortages of mathematics teachers in 43 States, critical shortages of 
earth sciences teachers in 33 States, and of physics teachers everywhere. 

• Half of the newly employed mathematics, science, and English ti3achers are no^: qualified 
to teach these subjects: fewer than one-third of U.S. high schools offer physics taught by 
qualified teachers, (pp. 22-23) 
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This publication focuses on one major area that has been identified by these reform initiatives, 
namely, teachers (especially vocational teachers) and their teaching effectiveness. 



Current Views from the Field 

Defining Ttachtr Effectiveness 

The quality of learning . . . depends very largely on the quality of the teaching— teaching 
that guides and inspires learning in the classroom, and that directs and moti*'ates learn- 
ing to be done in homework. Largely but not entirely! Effective learning often occurs in 
spite of defective teaching. Teaching at its best is only an aid to learning, but that aid is 
most needed by those who are least adept at learning. (Adier 1982, pp. 49-50) 

Any attempt to address the myriad of initiatives and agendas for reform of teaching must 
begin by first defining what is meant by "teacher effectiveness." According to Medley. Coker. and 
Soar (1984), teacher effectiveness is "the results a teacher gets; it is defined in terms of what pupils 
do, not what the teacher does or can do" (p. 15). 

W^.en applied to a group of teachers (for example, a school's faculty), teacher effectiveness 
forms the essence of what we call school effectiveness, it is an assessment of the impact of a given 
teaching/schooling process on the whole quantity and quality of student learning. !t implies an 
understanding of and consensus regarding exactly what learning— knowledge, attitudes, values, 
behaviors, competencies, and skills— is to be accomplished. 

In vocational education in particular, the question of the proper focus of learning has been 
disputed by the various audiences it involved: students, business and industry. State and Federal 
departments of education, the general public, and local boards of education. Furthermore, the 
n.Mure of their demands on vocational education and vocational educators has changed over the 
years. Present National emphasis on "getting back to basics" has spawned a new trend of teaching 
basics through vocational education. The complexity of this new role for vocational education has 
been compounded by continual demands to update vocational curricula in the latest technology, 
machinery, and working philosophies. In light of these demands, as well as Federal mandates for 
vocational education to respond to the needs of special and disadvantaged populations and to 
promote equity in occupational preparation, it is difficult to arrive at a uniform and consensual 
definition of the term vocational teacher effectiveness. 

Perhaps the first step in defining effectiveness for state- and local-level administrators and 
educators is to realize the demands being placed upon vocational teachers and to specify the 
areas in which improvement is most relevant. Thus, before assessment and improvement of either 
individual teachers or vocational teaching can begin, policymakers must identify the specific areas 
that warrant the efforts. 

At the level of the individual teacher, teacher effectiveness has traditionally been "operational- 
Ized" by taking the specific course objectives as the leariiing objectives to be addressed. This 
approach is not necessarify the most relevant one in vocational education inasmuch as many 
courses have an "unwrittet^ agenda" and teachers play a multitude of roles both in and outside the 
classroom. An enumeration of criteria for teacher effectiveness should take into account both the 
written and unwritten student learning objectives of a particular course. 
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Teacher effectiveness involves (1) the interaction of teacher activities with the learner; (2) all 
of the internal conditions that the learner brings to the interaction; and (3) any intervening contex- 
tual variables, such as the availability of teaching/learning aids, school structure, and other stu- 
dent inputs into the specific learning process. Although these interactive effects of a given teacher 
on a given learner are difficult to identify, the concept of probability gives vocational policymakers 
and practitioners a way to estimate teacher effectiveness by generalizing the effects of teachers on 
many students over time. 

We can, therefore, develop the following working definition of teacher effectiveness: 

Effectiveness of teachers and teaching is an assessment of the degree to which the 
majority of students can successfully learn a specific set of objectives from a given 
teacher. Further, it is an assessment of whether these objectives are both valid and rea- 
sonable and whether such results can be consistently achieved by a given teacher over a 
time frame (say 5 years). 

To the extent that the objectives and/or the students change over time, teacher effectiveness 
becomes a difficult, if not too obscure, concept to measure. If measuring "improvement" in teacher 
effectiveness over time is important, because of variations in teacher practices or teaching behav- 
iors, then maintaining as cunsiant a classroom and as consistent a set of learning objectives as 
possible is also desirable. 

Thus, at the individual level, it makes more sense to talk about teacher performance and 
teacher competence, unless one is doing research on effectiveness involving many observations 
and controls. These aspects of teaching are described and discussed in greater detail in chapter 3. 

From a broader, policy-oriented perspective, we can identify several factors that affect the 
Individual and collecMve effectiveness of teachers. These include the way in which teachers are 
prepared, credentialed. updated, and evaluated, as well as the way teachers are allocated within 
schools and distributed geographically and across teaching fields. Maintaining and improving 
vocational teacher effectiveness in this sense mean ensuring that the individual teachers are com- 
petent and able to perform well in classrooms, maintaining a sufficient quantity of qualified voca- 
tional teachers to meet ever-changing needs, and providing teachers and schools with adequate 
resources and support so that they may be most effective in promoting student learning. 



Teacher Supply and Demand 

Regardless of how competent individual teachers may be. teacher shortages diminish the 
overall effectiveness of vocational teaching. Although little firm data exist on the supply of p.nd 
demand for vocational teachers in general, continual shortages exist in certain occupational fields 
and in some geographical areas (the South and the West). Projections for the next 5 years indicate 
the overall demand for secondary teachers will decline during this period. However, beginning in 
the early 1990s demand for secondary teachers is expected to rise. In addition, teacher retirements 
are expected to create a large demand for new teachers. The National Center for Education Statis- 
tics (NCES) (1984) projects a need for more than a half million new secondary teachers by 1993 
(Janson 1985). 

Vocational teachers represent approximately 10 percent of the primary and secondary teacher 
work force and 19 percent of all secondary teachers. If one assumes that the demand for second- 
ary vocational teachers will be roughly proportional to the demand for all secondary teachers, then 
95.000 new vocational teachers will be needed by 1993. 

5 
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Meanwhile, another NCES study reports a rapidly increasing demand for vocational teachers 
at the postsecondary and adult levels during the 1970s. Although this rapid demand for teachers 
(40-43 percent growth over a 6-year period) seems to have subsided in the 1980s, growth con- 
tinues in some geographic and specialty areas. Craig (1985), for examole, has kept detailed infor- 
mation on the supply and demand for teachers of agriculture education for the past 20 years. 



Factors Affecting Ttachtr 
Quantity and Quality 

Many factors— not just one— affect the quantity and quality of vocational teachers. Some of 
these factors are discussed in the following sections. 

Occupational mobility. Vocational teachers often have skills much in demand in the labor 
market. Through their regular contacts with business and industry, these teachers become aware 
of employment opportunities, and employers become aware of their expertise. Many of these 
employers, being in the private sector, can offer higher salaries and benefits than school districts. 
When the teachers accept positions in the private sector, the result is an expensive loss to the 
school district of trained personnel. Fixed salary schedules that do not compensate for the inher- 
ently competitive market tend to have less success in attracting and retaining vocational teachers, 
particularly in areas of high demand. 

Working conditions. Working conditions and the organization of schools affect the quality of 
teaching and quantity of teachers. Some of the problems cited by practitioners include— 

• excluding teachers from decision making, 

• compartmentalizing teaching. 

• requiring heavy work loads (teaching and various administrative duties). 

• offering few or no professional development opportunities. 

• minimizing interaction among teachers, and 

• offering few or no rewards or recognition for good teaching. 

Systems that are not abie to address these problems and others like them give teachers little incen- 
tive or support for improving their effectiveness. Many teachers, understandably, decide to leave 
the profession. 

Preparation time. As a result of differing certification requirements and local shortages of 
teachers, many vocational teachers enter the classroom with inadequate preparation. Many voca- 
tional teachers enter teaching directly from occupational preparation and work experience with lit- 
tle or t^o teacher education or academic training. In such vocational areas as home economics, 
agricultural education, office occupations, and marketing, where professional, academic, and 
teacher preparation closely resembles that of nonvocational teachers, prospective teachers often 
have difficulty obtaining the appropriate work experience they need to teach. 

Some teacher educators consider inconsistent certification requirements the cause of this lack 
of standardized preparation. Others fault vocational teacher preparation programs that are 
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scattered among different colleges according to specialty areas, rather than centered in a compre- 
hensive program for all vocational teachers. Clearly, the total preparation time required to "fully 
qualify" an individual for vocational teaching is much greater than what people are able or willing 
to undertake, given the current return on the investment. Such preparation often requires occupa- 
tional licensing, receipt of teaching credentials, work experience, and possibly academic prepara- 
tton. Moreover, preparation and certification often fail to address many of the professional 
competencies ranked by practitioners as highly relevant. These competencies include maintaining 
student rapport, working with business and industry in cooperative programs and advisory com- 
mittees, recruiting students, and developing vocational programs. 

Instrvice teacher development Typical teacher supervision, evaluation, and staff development 
procedures offer little support for teachers to continue their training effectively. Specifically, the 
lack of time and resources for staff development is one difficulty many schools face. Af ;o, the tra- 
ditions of teacher isolation and assignment of work loads by seniority not only make improvement 
difficult, but also tend to promote the use of ineffective coping strategies. Even teachers who want 
to change find it difficult after years of such practices. 



Changing Demands on Vocational Education 

Vocational teachers are continually challenged to update and upgrade their teaching skills in 
order to remain effective. They must become familiar with technological changes in processes and 
equipment as well as learn to work with changing student populations and their special needs. The 
new requirement that they teach transferable skills, integrate basic skills, and prepare students for 
job clusters rather than specific vocations also demands that vocational teachers sharpen their 
skills. In some of these cases, teachers must retool completely in order to pursue these new 
challenges. 



Improvement Strategies 

In response to these challenges and problems that affect teachers and teaching, numerous 
actions and strategies have been proposed by various actors at Federal, State, local, and regional 
levels. Brief descriptions of some of the major efforts to change teaching and discussions on their 
particular relevance for vocational education follow. 



Recruitment and Selection 

Recruitment strategies are being aimed at reducing real or projected shortages in certain 
fields and increasing the quality of candidates overall. They are being designed to make the pro- 
fession more attractive and to overcome or eliminate obstacles or barriers to entry that prevent 
capable individuals from choosing to enter the teaching profession. 

A primary strategy— increasing teacher salaries— has been proposed or implemented in many 
areas. Other strategies are being used to improve the teaching environment, which helps retain 
good teachers as well as recruit high-quality initiates. A few such strategies are as follows: 

• Offering scholarships 

• Agreeing to forgive loans 
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• Developing programs that inform prospective teacher candidates about teaching 

• Extending teaching contracts to 1 1 months or 1 year 

• Providing career ladders and merit pay 

• Providing opportunities and resources for updating occupational and teaching skills 

• Allowing time for maintaining contacts with parents, community, and other teachers 

• Providing up-to-date equipment and materials that improve teaching practices 

• Increasing recognition and responsibilities of teachers 

• Improving working conditions 

Recruitment strategies that address the aforementioned teaching conditions (salary, work 
load, and so forth) have both short-term and long-term effects. In the long term, they can be used 
to attract a larger pool of untrained but capable individuals to the profession. Once a large pool of 
applicants is attracted, the challenge to vocational educators is to develop a mechanism for select- 
ing from among these talented individuals those most likely to become the best vocational 
teachers. Researchers who have looked at the relationship between academic ability (as demon- 
strated by grades and achievement tests) and teaching performance/effectiveness have not found 
a direct link. At best, academic ability provides a "minimum" criterion below which a teacher is not 
likely to be effective. 

Clearly, to improve their recruitment efforts, vocational educators need to develop better crite- 
ria for selection. Also, research is needed on the relationships among candidate attributes, success 
in teacher preparation, and occupational competence, as well as on their combined effect on the 
ultimate performance satisfaction of vocational teachers. 

Recruitment efforts may also address specific needs, such as meeting shortages of qualified 
trade and industry instructors. In this case, an effective strategy might be to identify and target 
specific groups for recruitment efforts. Such groups could include the following: 

• High school students interested in the shortage subject area 

• College students in fields other than teaching 

• Students/apprentices in occupational preparation programs 

• Teacher education students in other specialties 

• Practitioners in business and industry 

• Former vocational teachers 

• Former military personnel with training in vocational/technical areas 

Such strategies also imply that a specific preparation or development program is needed to equip 
the individuals in the targeted groups so they wif be professionally competent teachers. 
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Teacher certification is the process by which States protect the public from incompetent prac- 
titioners. Bratton (1984) points out that because certification is a legal requirement, and not simply 
a recognition of competence by a professional organization or agency, it should technically be 
called licensure. Whatever its name, the primary function of the certification process is to ensure 
the quality and competence of those who wish to enter teaching. A secondary function of licensing 
has been to regulate supplies of teachers to match demands by raising and lowering standards as 
appropriate. 

Teacher certification has traditionally involved a simple verification to ascertain whether a per- 
son has completed the prescribed training program and, in the case of vocational teachers, pos- 
sesses the requisite occupational license and work experience. As a result of increasing public 
pressure for accountability and quality assurance, many States have been upgrading their teacher 
certification and recertification requirements. 

The introduction of teacher competency tests and standardization of the baccalaureate degree 
as the minimum credential for full certification are prominent developments in the last 15 years. 
Yet these and other recent policies raise numerous unanswered questions and concerns such as 
the following: 

• How valid and reliable are the various criteria used to screen candidates and what is the 
actual impact of these criteria on improving teacher quality? 

• Is it wise to assess teachers* general skills and knowledge only after the professional 
preparation has been completed? 

• What are the real trade-offs between reducing teacher shortages and increasing certifica- 
tion requirements, the effects of various assessment criteria on concerns of equity, and 
the impact of increased certification requirements upon vocational teacher certification 
and effectiveness? 

After reviewing certification requirements for vocational teachers across the United States. 
Miller (1982) points out that there is "considerable variation of certification criteria between States 
and inconsistency in the standards within States" (p. 27). Despite the fact that bachelor's degrees 
are now required by all States in order to obtain a teaching license (Feistritzer 1983). most States 
have some mechanism— be it a temporary, provisional, emergency, or class IV certificate-- 
whereby teachers can actually begin their teaching careers without a bachelor's degree. 

Regarding the specific criteria for assessment, the report A Nation at Risk: The Imperative for 
Educatior)al Reform (National Commission on Excellence in Education 1983) recommends that 
"persons preparing to teach should be required to meet high educational standards, to demon- 
strate an aptitude for teaching, and to demonstrate competence in an academic discipline" (p. 30). 
With respect to vocational teachers in particular, the National Commission on Secondary Voca- 
tional Education (1984) recommends that "certification of all teachers should include both an aca- 
demic program and work experience record of demonstrated mastery in their field" (p. 10). 

Just how these broad recommendations are translated by States into specific measures of 
competence is uncertain as well as controversial. The uncertainty stems from the lack of a defini- 
tive body of research specifying the relationships among teacher characteristics, aptitudes, com- 
petencies, and attitudes/values and desirable learning outcomes in students. In this respect, 
Ornstein (1984) notes the following: 
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At present, we are unable to answer with confidence the influence a teacher has on stu- 
dent performance or behavior. The reasons are that the variables are too numerous and 
the interactions and relationships are too complex and multidimensional, (p. 109) 

The research on teacher effectiveness has verified the existence of identifiable and measur- 
able teaching skills that make a difference in student learning. Although this body of work still 
needs considerable expansion, particularly in the area of vocational education, it nevertheless pro- 
vides a starting point for developing potential criteria for teacher certification. Numerous reviews 
describe the esults of this research in detail (McGreal 1983). 

In addition to teaching competencies, certification of vocational teachers needs to identify the 
appropriate level of occupational experience and academic skills necessary for teaching voca- 
tional education. Occupational certification criteria should be based upon occupational analysis 
and should aadress all of the major competencies required by the job. Additional standards for 
certification should reflect the current state of knowledge about teaching and should not purport 
to measure competencies that are either unrelated to effectiveness or associated with a greater 
level of detail than research has confirmed as relevant. 

Determining certification criteria should be the responsibility of State departmerits of educa- 
tion and should be implemented in collaboration with teacher educators, local administrators, and 
teachers recognized by their profession. Although basic criteria should be the same for all 
teachers, assessment of occupational preparation should be flexible enough that individuals with 
different combinations of academic preparation and occupational background can be assessed 
fairly. Research on what competencies contribute to the effectiveness of vocational teachers in 
performing each of their multiple roles needs to be continued to support the development of valid 
and reliable certification criteria. 
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CHAPTER 2 

AN ACTION AGENDA FOR IMPROVING THE EFFECTIVENESS 
OF VOCATIONAL TEACHERS AND TEACHING 

This chapter translates the broad policy issues and concerns that were presented in chapter 1 
into specific actions that State legislatures, State departments of education (SDE) and vocational 
education (SDVE), and colleges and university departments preparing vocational teachers might 
undertake in addressing teacher effectiveness. The broad agenda is organized by agencies in 
order to give the reader easy access to the specific agenda that relates to his or her own role in the 
improvement task. These recommended actions are intended as a menu of strategies to be consid- 
ered by actors both individually and collectively in planning their own approaches for 
improvement. 



State Legislatures 

State legislatures can make a great contribution to the issue of teacher shortages. As dis- 
cussed in chapter 1, the problem of teacher shortages is not easily solved by simple solutions. 
State legislatures can play a leadership role, however, by commissioning statewide surveys of the 
nature and extent of teacher shortages and surpluses in specific teaching fields. They can also 
request statewide projections of need for vocational teachers, define the role of vocational educa- 
tion at the secondary and postsecondary levels, and project, based upon these goals, the need for 
vocational teachers with specific qualifications in each occupational area. 

State legislatures play a critical role in setting uniform teacher certification standards. Their 
role includes bringing together information on such topics as (1) the reality of teacher shortages 
and public demands for quality assurance, (2) teacher educators* inputs into the standards that 
realistically can be set, and (3) State and local administrators* perspectives on implementation. 
State legislators are also able to call on their business and industry contacts to assist in planning 
and developing local inservice programs for teachers. Finally, State legislatures can allocate funds 
for the following: experimenting with alternative forms of vocational teacher preparation; support- 
ing exchange of information for program improvement; and conducting school-based research 
and development to improve learning, teaching, and supervision. 



State Departments of Education and 
Vocational Education 

At the State level, SDEs and SDVEs perform multiple roles in ensuring and supporting the 
effectiveness of teaching and schooling. Although State and Federal legislatures need to appro- 
priate funds for educational improvements, it is the role of SDEs and SDVEs to ensure that these 
funds are used to respond to the legislative mandates. It is their role also (O provide information 
about the nature of education and to offer guidance and leadership in proposing new directions 
for improvement to the legislature, the public, and the educational community. 
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Teacher Shortages and Surplum 

State departments of education must act to alleviate present teacher shortages and to avert 
future shortages and surpluses. Short-term solutions to shortages, such as recruiting teachers 
directly from business and industry or substituting teachers who are not certified to teach in a par- 
ticular area, are stopgap measures at best. 

in the immediate future, SDEs and SDVEs can support preservice and inservice activities 
directed toward developing teachers in areas of critical need. For example, they can— 

• develop scholarship programs targeted to those already in the university system or other 
areas of teaching to teach in critical shortage areas and 

• assist (provide incentives for) the some 15*20 percent of nondegreed vocational teachers 
in completing degree programs and becoming fully certified teachers. 

In the longer term, SDVEs can use the information they now collect on teachers to project 
future demands for new toachersand to assess overall teacher development and upgrading needs. 
Where shortages are anticipated, SDVEs can work with State legislatures regularly to ensure more 
competitive salaries for teacher's in general and to suppon school improvement programs that 
make the profession more attractive, thereby ensuring a steady supply of teacher candidates. Thoy 
can also help institutions preparing vocational teachers to keep track of labor supply and 
demands. Finally, they can ensure that adequate labor market information is provided to prospec- 
tive vocational teachers, thus enabling them to make informed decisions about their careers. 



Certification and Rtcertification 

Other key roles of SDVEs are developing and implementing certification standards for 
teachers and developing accreditation standards for vocational programs and teacher preparation 
programs. These standards and their implementation provide the public with an indicator of how 
well their tax monies are being spent. They are one means by which the education community can 
elicit future support for educational programs. The responsibility of SDVEs and SDEs to make 
these standards acceptable and comprehensible to the public needs to be balanced with their abil- 
ity to set valid and reliable standards and to collect information on their use. SDVEs also have an 
implicit responsibility to the profession to establish standards that will help administrators and 
instructors in the process of improving the effectiveness of teaching. Moreover, vocational educa- 
tion should have uniform assessment procedures for graduates of teacher education institutions, 
as well as for individuals coming into tcraching directly from business and industry. 

Although the specific criteria and means of assessment naed to be determined and validated 
on a state-by-state basis, it is generally agreed that prospective teachers should demonstrate pro- 
ficiency in basic skills, their subject-matter specialty, general knowledge, and pedagogical princi- 
ples. Incentives should be built into the certification process to encourage nondegreed vocational 
teachers to upgrade their education and complete a degree. Prior to oNaining full certification, 
new teachers should participate in an induction period of at least 1-3 years, during which time 
their development would be carefully monitored and supported. At the end of the induction period, 
an assessment of their teaching performance would be made. 

In addition to certification of secondary school vocational teachers, postsecondary school 
instructors also should be certified by SVdte departments of education. Posisecondary community 
colleges and technical institutes should implement planned programs of staff development leading 
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toward certification for both full-time and part-time instructors. The use of an individualized and 
modularized delivery system, such as that which has been implemented in the State of Arkansas, is 
recommended. This statewide system is also a competency-based teacher education system based 
on the performance-based teacher education (PBTE)modules developed by the National Center 
for Research in Vocational Education. 

At a time when the knowledge base about teaching and learning is expanding and the techno- 
logical basis for occupational education is ever changing, vocational teachers need to be continu- 
ally updated in their craft and in their profession. To ensure that this updating occurs, teachers 
should be recertified periodically. In addition. States need to update specific certifications and 
recertification criteria at regular intervals. 



Teacher Preparation Programs 

In addition to monitoring teacher education programs. SDVEs need to collect information on 
the success of graduates in their eariy teaching experiences. A number of States already collect 
information about teacher performance and credentials .n their Management Information Systems 
(MIS). However many States do not analyze these data to assess the effectiveness of various 
teacher preparation programs. 

Through their Federal program improvement func 'Es can support various types of activi- 
ties that will enhance teacher preparation programs. For example. State departments might pro- 
vide seed money for developing new programs for updating and upgrading teacher educators. 
Additionally, they might fund pilot programs that use new forms of deliver/ or media for teacher 
preparation such as field-based, individualized, and/or modularized programs. In States where a 
large percentage of vocational teachers are nondegreed or come directly from business and indus- 
try. SDVEs should provide leadership and support for initiatives ensuring that these individuals are 
adequately prepared to teach before they enter the classroom. In any case, these initiatives should 
addresp. the identified priority needs of the State, 



School-Based Improvemtnt 

The evidence on effective schools and teaching, as well as research on the change process, all 
cleariy indicates that the initiative and impetus for school improvement ultimately resides at the 
local school level. In addition to the activities already mentioned. State departments of vocational 
education can support school-based improvement of vocational teaching by doing the following: 

• Supporting the development of new approaches to formative evaluation that will enable 
schools and teachers to identify improvement needs and develop individual teacher goals 
more effectively 

• Synthesizing and disseminating current research findings on teacher effectiveness and 
supporting their application to vocational teacher evaluation 

• Providing leaderahi;^ and support for teacher inservice and career development activities 

• Facilitating closer collaboration and linkages among colleges of teacher preparation and 
local school districts 

• Supporting Tield-based research that will add to the knowledge and information base on 
the effectiveness of vocational teachers in their various roles 
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Finally, increasing sophistication and technical expertise are being demanded by State 
department staff in providing data, information, analysis of problems, leadership, decision making, 
problem solving, and communication and persuasion. Developing and upgrading staff knowledge 
and skills should be a high priority for SDVEs. Many SDVE administrators are recruited from uni- 
versities and colleges specializing in teacher preparation. Teacher educators moving to adminis- 
trative positions in State departments of education should undergo inservice training to sharpen 
managerial, problem-solving, and decision-making skills. 



Teacher Preparation Institutions 

Numerous actions may be undertaken by colleges and universities that prepare vocational 
teachers. The recommendations are organized into the following five broad areas: 

• Recruitment, selection, and screening of students 

• Program and curriculum 

• Quality of faculty 

• Evaluation of programs 

• Outreach activities 

One additional function that is generally assumed by colleges and universities is the conduct of 
educational research. The recommendations pertaining to this role of teacher educators are 
implicit in each of these topical discussions. 



Rtcrultment, Selection, and Screening of Students 

The recruitment, selection, and screening procedures used in a teacher education program 
are as important in determining the quality of teachers as is the quality of the program itself. Until 
now, the salary and conditions of teaching have generally failed to attract the most talented stu- 
dents into the profession. Teacher preparation programs have played, at best, a passive role in the 
selection and screening of teacher candidates. With States moving to increase teacher salaries, 
and with the educational reforms aimed at improving the quality of teaching, the screening and 
selection of teacher candidates will become a critical factor in ensuring the quality of teachers. 
Therefore, teacher education will need to ensure that the criteria and standards for admitting and 
retaining students in teacher preparation programs are reliable and have been validated. Criteria 
might focus on the chartcteristics and qualities of prospective students that foretell successful 
performance (1) in the teacher education program and (2) in a future teaching position. 

It might be useful here to mention the need to distinguish between the criteria for selecting 
individuals preparing to teach at the secondary and postsecondary levels. In light of the recent 
movement toward redefining the role of vocational education at the secondary level, the general 
skills and occupational knowledge and experience required may be quite different for individuals 
preparing to teach at these two levels. The sections on competency testing and teacher interviews 
In chapter 3 of this publication offer several specific suggestions for the screening and selection of 
teacher education candidates. 
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In areas of current and projected teacher shortages, teacher education institutions can exert 
significant influence in recruiting prospective teachers simply through on-campus activities. They 
might provide information about scholarships and other State or locally sponsored activities to 
attract prospective teachers. They might recruit able students and graduates from other programs 
by providing information about the teaching profession, local training programs, and teaching 
opportunities in their respective States. In this regard, colleges and universities must work closely 
with SDEs and SDVEs to ensure an accurate and comprehensive picture of current and projected 
teacher needs. Through active outreach in local schools and with business and industry, faculty 
members can provide potential teacher recruits with realistic and current information about what it 
means to be a vocational teacher. 



Programs and Curriculum 

Because of the diverse learning needs of students in vocational teacher preparation, the 
teacher training curriculum must, first and foremost, be flexible. Students have a wide range of 
educational, training, and experiential backgrounds. For example, some may hold baccalaureate 
degrees with little or no teaching experience. Others may have extensive subject-area experience 
but only a high school certificate. Still others may have some teaching experience with little or no 
formal teacher preparation. Preparation programs must address learners' individual needs, while 
ensuring that graduates obtain a comprehensive set of skills and knowledge. The use of diagnostic 
testing procedures to ascertain the leva! and breadth of students' previous learning and cognitive 
learning style, in combination with counseling, may be useful in creating individualized develop- 
ment plans. 

The curriculum should be designed so that teacher preparation is seen as a continuous 
process— including both preservice and inservice components. Exposure of the uninitiated to the 
experience and working conditions of teaching eariy in the program will provide realistic expecta- 
tions and will promote efficient learning. Although eariy exposure and sequenced practicums inte- 
grated throughout the course of study are useful learning tools, care must be taken to assess the 
relevant knowledge base of the student prior to any performance assessment. 

The core curriculum or knowledge base needs to be examined, evaluated, and revised contin- 
ually in light of evolving research on learning and effective teaching practices, and other changes 
in the content area. Teacher educators need to adopt the most effective practices and serve as role 
models to prospective teachers. They should explore new learning technologies, such as individu- 
alized learning modules and videotaping, and integrate them into teacher education programs as 
appropriate. 

Teacher educators should consider linkages between teacher preparation and administrator 
preparation programs, particulariy during the practicum experience. Such early collaboration can 
potentially establish new, facilitative roles and relationships. 

Besides being flexible in meeting the needs of students from various backgrounds, the teacher 
education curriculum must also provide prospective teachers with the tools they need to teach 

(1) in a variety of settings (e.g., secondary, postsecondary, and business and industry settings). 

(2) a variety of learners (children, youth, adults, and ''special" populations), and (3) a variety of 
subject areas. Because the unique aspects of the individual specialty areas of vocational education 
must be addressed when preparing vocational teachers, teacher educators from all of these spe- 
cialty fields should work closely together to develop a comprehensive program that includes a 
basic core curriculum succeeded by alternative specializations. 
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There is currently great diversity in the quality of vocational teacher preparation programs. 
This is partly due to the tact that, in some cases, teacher preparation has been linked historically 
to professional development, whereas other p.^-^rams have had no such support. Some vocational 
educators have c&lled for development of compr0h0nsiv0 vocational education departments that 
would place all of the vocational teacher preparation programs under one roof. 

In earlier discussions in this publication about programs and curriculum, a comprehensive 
vocational teacher preparation program with multiple specialization options was proposed. Cen- 
trisHzing vocational teacher preparation within colleger and universities could both reduce the 
variation in quality amor^g programs and save individual department resources. These resources— 
human and material— might be more wisely spent in updating and upgrading teaching programs 
and in furthering the research base on effective teaching. College and university administrators 
should bring together teacher educators from all vocational areas and from general education to 
determine appropriate delivery mechanisms for all areas of vocational, technical, and general 
education. 

Given a comprehensive vocational teacher education program, institutions could be more 
responsive to projected teacher shortages and surpluses in specific occupational and career areas. 
By preparing teachers for a broad range of settings— for example, vocational schools, high 
schools, postsecondary institutions, and business and industry— survival of a compre^iensive pro* 
gram would be less dependent upon fluctuations in demand for its graduates in any single setting, 
thereby ensuring both stability and continuity. Such security would enable institutions to set and 
maintain high-quality standards. 

Although political and historical realities at some teacher preparation institutions may make 
this course of action a long-term possibility at best, recent proposals to place te&chei education 
programs entirely at the graduate school level may provide a new impetus for such a change. 
Proponents suggest that undergraduate preparation should focus on a liberal arts education and 
on the development of general an'J subject-area skills and knowledge; graduate education should 
focus on the development of pedagogical competencies. One drawback to such a plan is that it 
would put nondegreed vocational teachers at an even greater disadvantage than they have been in 
the pa^t. Nevertheless, the current attention being given this idea will continue and will cause 
vocational educators to reexamine the natura and delivery of future vocational teacher 
preparation. 



QuaUty of Faculty 

Teacher educators should use their own teaching to demonstrate effective instructional 
theories and practices taught in the teacher preparation curriculum. They should provide prospec- 
tive teachers with models of the processes and practices they are advocating. 

Promoting the quality of teacher educators involves the following: 

• Ensuring adequate screening of candidates for entry into the profession 

• Rewarding those practices in teaching, research, and community service that contribute 
substantially to the field 

• Providing opportunities for staff to develop new skills and update their knowledge 
e Evaluating teaching performance pericaically 
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Since the evaluation of teacher educators is a complex process, more control should be exer- 
cised at entry into the profession as a quality assurance measure. For example, an assessment 
center mighl evaluate candidates seeking admission into programs that prepare future teacher 
educators. 

Ongoing, systematic, staff development programs should be offered for teacher educators. 
Strategies to upgrade teacher educators might include— 

• providing internships for all new vocational teacher educators to familiarize them with 
"the field." 

• developing visiting scholar programs that periodically place teacher educators into public 
schools for a period of time. 

• requiring some form of upgrading as a condition of continued employment, and 

• collective bargaining to ensure faculty participation in staff development activities. 
Some areas in which teacher educators need staff development include the following: 

• Dtvtlopmtnt of consulting tklllt— School districts and other agencies are increasingly 
seeking professional assistance from teacher educators. 

• Provision of technological updating In ttitir occupational fields— Rapid technological 
changes in many vocational areas require vocational teachers and teacher educators alike 
to keep abreast of new developments. 

• Insights Into the business and Industry perspective of training and development 
sptclallsts— The preparation of human resource developers (HRDs) for business and 
industry is a role that teacher educators have neglected. Activities such as becoming 
directly involved in community activity, participating in HRD professional organizations 
and activities, and reading relevant HRD professional literature would enhance teacher 
educators* breadth in this. 

• Development of a shared philosophy— Faculty should develop a shared understanding 
and philosophy of the teaching and learning process, their role as vocational teacher 
educators in colleges and universities, and the roles and contributions of vocational edu- 
cation in schools and of schools in society. 



Evaluation of Programs 

Periodic evaluations of teacher preparation programs should be undertaken by both faculty 
members and external reviewers. The evaluation agenda should explore outputs (how successful 
the program is in preparing teachers for real jobs in the field), process (how well the program is 
functioning), and inputs (whether the goals, objectives, and resources are appropriate and 
sufficient). 

Useful information about the effectiveness of the program in preparing vocational teachers 
might be ascertained from— 

• SDEs. if they obtain information on teacher performance and transcripts of teachers cur- 
rently employed; 
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• follow-up surveys of graduates and interviews with their supervisors to identify perfor- 
mance difficulties and needs for specific kinds of knowledge and skills; and 

• direct assessments of the early teaching experiences of graduates, which might be facili- 
tated by active faculty involvement in school-based activities, such as participation in 
classroom observations and in support and advisory teams. 

The evaluation of teacher education programs should be linked to a strategic planning pro- 
cess whereby new initiatives at State, regional, and National levels are examined along with pro- 
gram operation and effectiveness data. This process would identify and analyze future policies, 
practices, and directions for teacher education programs. 



Outreach Activities 

Because of their rote as educators and researchers, the faculty of teacher preparation pro- 
grams, particularly faculty in vocational education, support improvements in vocational teaching 
through a variety of functions beyond that of teaching preservice teachers. In many States, teacher 
trainers in vocational education traditionally have provided extensive "field services" to local 
school districts despite strong institutional disincentives for so doing. Projections of an increas- 
ingly "aging" population of teachers over the next 15 years will require an increased propc: Won of 
field-based inservice vocational teacher training. Additionally, the proliferation of induction-year 
programs for intern teachers provides new opportunities for teacher educators to participate in the 
teacher inservice, support, and assessment process. Teacher educators must be flexible and able 
to respond to current and developing needs. 

Finally, teacher educators should work with their SDVEs in developing ccTiprehensive criteria 
for assessing graduates from teacher education programs prior to employment. Criteria should 
similarly be developed for part-time or temporary teachers from business and industry that would 
ensure and facilitate the transfer of their specialized skills and technical knowledge. 
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CHAPTER 3 
TEACHER EVALUATION STRATEGIES 



Teaching is a complex transaction influenced by many variables extraneous to the process 
itself. Despite years of focusing on this activity, researchers are still unable to st^ite with a reason- 
able degree of confidence exactly what constitutes good teaching or which attributes lead to 
teacher effectiveness. The problem is further confounded because teaching and learning form one 
entity. Not only are vocational researchers uncertain about the teaching process, but they also 
lack understanding of human learning. 

For all these reasons, it is not surprising that no simple evaluation can be readily applied to 
sort out poor teachers from good ones. Because of the complexity involved in assessing teacher 
effectiveness iw a valid, reliable, objective, and fair manner, various strategies have been adopted 
over the years to accomplish this task. 



The Need to Focus Ttachtr Evaluation 

A typical approach to teacher evaluation in the past has been to visit the classroom, armed 
with a checklist, and check off items as the teaching process unfolds. This approach has little cred- 
ibility. It is degrading for the teachers and serves no useful purpose. A more systematic approach 
to teacher evaluation involves examining teacher evaluation efforts from the perspective of who 
does what, to whom. when, and for what purpose. 

The who variable can be a generator or collector of data and can include the following actors: 

• State departments of education 

• Teacher training institutions 

• School district administrators 

• School administrators 

In focusing evaluation efforts, it is necessary to determine when input from these individuals is crit- 
ical for making the most accurate assessment of teacher competencies. 

The what variable refers to the various strategies available for evaluating teachers. Some may 
be more reliable than others, and more important, some may be appropriate for specific purposes. 
Following are some of the common strategies in current use: 

• Competency testing 

• Interviews 

• Classroom observation 
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• Student ratings 

• Peer review 

• Self-evaluation 

• Student achievement 

The purposes of the evaluation effort ordinarily will determine which types of evaluation strategies 
are appropriately applied either individually or in combination. 

The type of teachers being evaluated must be considered when addressing the to whom varia- 
ble. Some examples of types of teachers in vocational education include these: 

• Individuals from business and industry seeking admission into teacher education 
programs 

• Individuals from business and industry seeking direct employment in vocational teaching 

• Graduates of teacher education programs 

• Nondegreed vocational teachers 

• Part-time teachers 

• Full-time teachers 

• Probationary teache s 

• Tenured teachers 

• Master teachers 

• Teachers returning to the profession 

This is one way of listing teachers by the status dimension. Other listings could be organized by 
subject matter or by types of students taught. 

When evaluation is performed is important. Evaluation can bP performed at various strategic 
points, which, in turn, may influence the frequency of evaluation. The following are some critical 
points at which evaluation is warranted: 

• At entry into the profession 

• At entry into teacher training programs 

• After exit from teacher training 

• Upon application for employment 

• After the probationary stage 
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• After the certification stage 

• After the recertification stage 

• Before promotion 

• Before termination 

• Upon identification of performance deficiencies or discrepancies 

The for what purpose variable addresses the justification for teacher evaluation efforts. Exam- 
ples in this category include the following: 

• To control entry into the profession 

• To make admission to teacher training program selective 

• To exercise quality control of graduates exiting from teacher training institutions 

• To recruit and select employees 

• To aid in certification 

• To aid in recertification 

• To justify withdrawal of individual certification 

• To aid in personnel decisions 

• To identify improvement needs 

• To reward exemplary performance 

These key questions— who does what, to whom, when, and for what purpose— provide a useful 
framework for focusing teacher evaluation efforts. In addition, such other contextual factors as 
organizational climate, class size, characteristics of student pop*jlations, and available resources 
can, given adequate consideration, improve the effort and provide more valid, reliable results. 

The remainder of this publication focuses on each of the approaches mentioned in the discus- 
sion of the what variable. Each individual discussion will include the following: 

• A description of the approach 

• Major issues in its use 

• Its advantages and disadvantages 

• Conditions for its use in conducting vocational teacher assessments 
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TMChtr Competency Testing 

Teacher competency testing is a broad-based approach to teacher evaluation. Three major 
purposes are typically served by teacher competency testing: (1) selective admission to teacher 
education programs, (2) teacher certification, and (3) teacher promotion. 

In teacher competency testing for selective admission, teacher education institutions assess 
prospective candidates seeking admission based on indicators that predict successful program 
completion and a successful career in teaching. Various prerequisites may be required in this 
"gatekeeping** operation, including passage of prescribed tests. 

Teacher competency testing for certHication refers to the assessment of the knowledge, skills, 
or behaviors judged as being essential minimum requirements for performing specific teaching 
tasks under specific conditions at minimum and predetermined levels of mastery. Teacher compe- 
tency testing for promotion is similar to testing for certification. It is the process used to assess the 
potential of individual teachers for success in performing high levels of tasks and responsibilities 
in a career hierarchy or teaching career ladder. 

The new wave of teacher competency testing has emerged mainly from the recommendations 
of major National reports and in response to reports of sharp declines in teacher quality and paral- 
lel, significant declines in student achievement on National standardized tests. This interest in 
teacher competency testing is also attributable in part to the results of public opinion polls. These 
polls have consistently evidenced a growing concern among American parents for teacher quality 
and have also indicated the strong support of Americans for teacher competency testing. On this 
topic, Ishler (1965) notes the following: 

Competency testing of prospective teachers seems to be taking the country by storm. It 
is being viewed both as a quality assurance measure for the general public and as a way 
of demonstrating that teaching is indeed a profession since other professions already 
require successful completion of an examination prior to entry, (p. 27) 

Sandefur (1965) notes that "the early impetus to assess the comp Jency of teachers was 
focused in the southeastern states** (p. 3). and that the movement was rapidly expanding north- 
ward. He also indicates that 30 States have already implemented teacher competency assessment 
and that another 12 States are in the planning stages. 

It is noteworthy that not all States are using the same mechanisms for initiating teacher com- 
petency testing. Twelve States have resorted to legislation for mandating teacher testing, while 22 
other States have chosen to issue SDE regulations requiring testing. The States of Arizona, 
Arkansas, and New Mexico have issued both SDE regulations and legislative mandates for teacher 
competency testing. Five States have mandated testing for admission to teacher education pro- 
grams only. Thirteen Spates have mandated testing for certification purposes only, and 12 States 
use testing for both admission and certification. 



Compttency Testing for Selective Admission 

Growing evidence indicating that the teaching profession is primarily attracting less able stu- 
dents, and not enough of the brightest students, has encouraged policymakers to propose legisla- 
tion to control the quality of students entering teacher education programs. The National Center 
for Educational Information (1984) surveyed teacher education institutions throughout the Nation 
and found considerable diversity regarding admission requirements and entry procedures. The fol- 
lowing type of admission criteria were reported by the institutions surveyed: (1) high school class 
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rank. (2) high school grade point average^ (3) Scholastic Aptitude Test (SAT) scores, (4) American 
College Test (ACT) scores, (5) college grade point average, (6) personal letters of recommenda* 
ticn. (7) interviews, and (8) prior experiences working with children. Goertz and Pitcher (1985) 
report that 17 States are using one or more of the following tests to screen candidates for admis- 
sion to their teacher education programs: (1) the Scholastic Aptitude Test (SAT), (2) the American 
College Testing Program (ACT), (3) the California Achievement Test (CAT), (4) the Pre- 
professional Skills Test (PPST), (5) the National Teacher Examination Programs Core Battery, and 
(6) State-developed examinations (p. 3). 

IMUM in UM. The major issue in using teacher competency test results for admitting or reject- 
ing candidates to teacher education programs is that of equity. A significantly high percentage of 
minority students are failing these tests and are thus being excluded from the teaching profession. 
This failure rate considerably reduces the representation of minority groups in the teaching 
profession. 

According to the National Commission for Excellence in Teacher Education of the American 
Association of Colleges of Teacher Education (AACTE) (1985), there is a projected need for 1 mil- 
lion new teachers by 1990. One in every 10 colle^ie students will have to enroll in teacher educa- 
tion programs to meet this National demand. At tne same time, the more stringent criteria for 
admission to teacher education programs will most likely have an adverse impact on enrollments 
and will accentuate the projected shortages of teachers. 

In vocational education, selective admissions will affect teacher education programs imme- 
diately, especially in the trade and industry areas where skilled craftsworkers and technicians are 
usually hired directly from business and industry with little or no previous teaching experience. If 
selective admission criteria are applied in trade and industry teacher education programs, many 
potential candidates will be screened out of programs. This raises the serious question of how 
academic accompHshments. such as SAT or other standard test results, should be weighed relative 
to occupational competencies, work experience, pedagogical skills, and teaching knowledge. The 
stringent application of highly selective admission requirements or teacher competency test 
scores will certainly reduce the pool of prospective vocational teachers, and the greater shortages 
will become a major issue in vocational education. 

Another major issue in the use of competency testing for selective admission is the absence of 
empirical evidence that individuals with higher SAT or other standardized competency test scores 
will make better teachers. Research in the field of cognitive psychology indicates that there is a 
strong correlation between individuals* cognitive style and their vocational choices. It has been 
found that the field-independent person opts for professions dealing with abstract cognitive think- 
ing such as engineering, whereas the field-dependent individual prefers social services occupa- 
tions. Given this natural self-selection process, can colleges of education compete for students 
from engineering? Will these students benefit from teacher training? Will they make better 
teachers? Will they be satisfied with their jobs? These are some of the questions that are now 
being raised, but that have no immediate answers. Also, when considering the implementation of 
teacher competency testing and selective admission policies, it must be remembered that research 
has time and again failed to provide any consistent empirical evidence of a positive relationship 
between teacher qualifications and students* learning. 

Advantages of compttency testing for stlectlve admittion. Among the major advantages of 
competency testing are the following: 

• Publicizing test results can contribute to an improved teacher competency image for 
teacher education programs, teacher education program graduates, and the teaching pro- 
fession in general, and it can restore and build public confidence in education. 
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• if teacher education students possess high levels of skills when admitted to programs, 
then higher levels of excellence in program achievements could be the target of instruc- 
tional programs. 

• Competency testing and selective admission have the potential of excluding individuals 
with low likelihood for success in teacher education programs and in teaching. When it 
works as it ideally should, competency testing can eliminate the waste that would be 
encountered in individual and institutional resources and the frustration that would occur 
if weaknesses were detected only during or after completion of teacher training programs. 

Disadvantages of comptttncy testing for Miectlve admission. Among the major disadvantages 
of competency testing are the following: 

• Minority students are disproportionately failing these tests and are being excluded from 
the teaching profession. One educator predicted that if this trend continues, "within the 
decade, the minority teaching force will be less than 5 percent, compared to 12 percent in 
1980" {Education USA. 30 July 1984). 

• The validity and reliability of competency tests are difficult to establish with any degree of 
precision. 

• Selective admission to teacher education programs can create an acute shortage of 
teachers, resulting in a larger class size and an inflationary wage effect. 

• Arbitrary and specific cutoff scores can eliminate professional judgment in decision mak- 
ing (Steadman 1984). 

• The complex nature of aptitudes required for teaching is reduced to qualities measurable 
by scores on standardized tests. 

• Tests are costly to purchase or develop and to administer. 

Conditions for um In vocational teacher training. The primary objective of vocational educa- 
tion is to prepare s^.udents for entry-level employment in business and industry. Therefore, voca- 
tional teachers must be masters in their occupational specialties, as well as teaching professionals. 
Consequently, when selecting students for teacher education programs in vocational education, a 
balance must be maintained between high levels of cognitive abilities and high levels of trade 
competence. Setting either academic or occupational requirements arbitrarily high might have the 
adverse effect of excluding competent tradespersons or academically talented individuals from 
vocational teacher education programs. 

Major National study commissions have recently made recommendations for strengthening 
the basic skills competencies of high school students, including vocational education students. 
This recommendation is also reflected in the Carl D. Perkins Vocational Education Act of 1984, 
which emphasizes the need to improve the academic foundations of vocational students. These 
developments signal the fact that tomorrow's vocational teachers will need a foundation in the 
basic skills, in addition to trade competence and experience. However, as noted earlier, in some 
specialties in the trade and industry areas in particular, individuals with such preparation may not 
be readily available. To deal with these kinds of problems, it may be necessary to develop remedial 
programs in basic skills and integrate them into the initial preparation of prospective vocational 
teachers. 
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Another consideration that makes selective admission a difficult undertaking is the rapid and 
ever-accelerating change in technology that calls for flexible and adaptive vocational teachers. As 
technology changes ever more rapidly, vocational teachers must demonstrate the flexibility and 
willingness to update tht^ir knowledge and skills frequently. The same is true with regard to teach- 
ing and learning theories and instructional technology. The changing emphasis in traditional 
teaching methods courses toward a research-based approach will make student teachers greater 
consumers Oi educational research and development. 

Since occupational competency and occupational experience are also of critical importance 
for vocational teaching, selective admission procedures should assess the prospective vocational 
teachers on both of these components. The National Occupational Competency Testing Institute 
(NOCTI) has developed a series of tests for assessing the trade competency of vocational 
teachers, called the Teacher Occupational Competency Tests (TOOT), that can be used as screen- 
ing devices for program admission. In some teacher training institutions, where passage of the 
TOCT is part of the admission requirement, advanced standing credits are given to students who 
are successful on these tests More details about the TOCT are provided later. 

Strong arguments exist in support of adopting selective admission policies for vocational 
teacher preparation programs. However, because of the specialized nature of vocational instruc- 
tion in which occupational competency and experience are so crucial, flexibility in admission 
procedures is necessary. Remedial training in general education and pedagogical skills may need 
to be a part of an initial inservice program for some new candidates in vocational teaching. In the 
process of developing and applying selective admission procedures to vocational teacher training 
programs, policymakers must guard against arbitrarily genernlizing and applying procedures devel* 
oped for other areas of education to vocational education. The danger of excluding large numbers 
of minority students from programs by the use of some kinds of competency tests also necessi- 
tates the consideration of additional and alternate routes to improved vocational teacher quality. 



Competency Testing for Certification 

Overview. Teacher testing for certification purposes is not an innovation. The practice dates 
back to colonial times when written and oral examinations were administered to prospective 
teachers. However, it has attracted new and widespread attention and support because of the find- 
ings and recommendations of major National commissions on education regarding recent declines 
in teacher quality. 

In her introduction to a recent article on teacher education, Denham (1985) notes that 

a recent loss of confidence in universities as providers of well-educated graduates, often 
coupled with suspicion that university teacher training programs are neither rigorous 
nor effective, has led about half of the fifty states to require state competency tests for 
the credentialing of teachers, (p. 41 ) 

The certification and licensing of civil servants are two screening strategies designed to 
ensure that individuals possess the required competencies for practicing their professions. They 
also protect from professional malpractice that can endanger public health, safety, and welfare. It 
is not surprising, therefore, that competency testing for teacher certification has gained rapid 
acceptance nationwide. Competency testing is a crucial element in teacher certification because 
teachers have almost absolute freedom and protection behind closed classroom doors. Moreover, 
the developmental stages of the students, and compulsory school attendance laws, make them 
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especially vulnerable to the judgments and decisions of adult teachers. Parents and taxpayers 
have the right to be certain that children are educated by competent professionals who can pro- 
vide the best and safest educational experiences. 

Roth (1985) defines teacher certification as "the process by which an individual is licensed to 
teacl> in a particular state** (p. 7). He goes on to describe some of the key implications and issues 
related to teacher certification as follows: 

In the United States it is mandatory that children within a certain age group attend 
school or have appropriate instruction. Along with this is the responsibility for the state 
to provide a high degree of assurance that school personnel are at least minimally quali- 
fied in order to instruct children in the classrooms or to perform other professionally 
related services in the public schools It (teacher certification) is a measure of protec- 
tion for the public to ensure that students receive what they are entitled to in terms of 
appropriate learning "ind growth experiences. State teacher certification is a means of 
protection to prevent the public from being exposed to either incompetent or immoral 
(legally defined) individuals who could potentially do irreparable harm to students, both 
intellectually and emotionally, (p. 0) 

Most teacher competency tests presently in ure consist primarily of paper-and-pencil tests, 
which are usually aimed at measuring some combination of prospective teachers* competencies in 
basic skills, subject-matter expertise, and pedagogical knowledge. More recently, an increased 
interest has developed in assessing the actuai performance of teachers in classrooms. The State of 
Georgia has been one of the front-runners in this movement with its Teacher Performance 
Assessment Instruments (TPAI). 

Some States require their vocational teachers to take trade competency and trade certification 
examinations in their occupational specialties in addition to general skills tests. These tests usually 
assess both the knowledge and performance capabilities ot teachers. 

According to Goertz and Pitcher (198^). 10 States are using the National Teacher Examination 
(NTE) Core Battery for teacher certification, 10 other States are using the NTE Specialty Area 
Tests for certification, and 6 States use their own locally developed certification tests. The TOCTs. 
which were developed by NOCTI. as well as some locally developed tests, are also being used by a 
few States to test vocational teachers* skills in their occupational area as part of their certification 
requirements. 

The National Evaluation Systems (NES) i? a private organization that has also been very active 
in teacher competency testing. It has orovided a cumptete range of teacher certification testing 
services to State education agencies since 1972. 

NES has developed over 250 teacher competency tests in more than 90 content areas, 
including professional and basic skills. All these tests are criterion-referenced. NES also provides 
assistance to State agencies in establishing the validity and reliability of all its tests, as well as in 
administering, scoring, and reporting the test results to its clients. 

Many States are now using teacher competency tests to phase out lifelong certification and 
phase in conditional, periodic reasii^ssment and recertification of teachers based upon continueo^ 
performance, growth, and development ov**r time. In those occupational areas of vocational edu- 
cation where technology is expanding rapidly, trade competency tests could be used for periodic 
reassessment of knowledge and skilb and recertification of vocational teachers. 
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iMuts in UM. Many of the issues already discussed in relation to teacher competency testing 
for selective admission are also of concern in the context of testing for certification. According to 
the Educational Testing Service (ETS) (1983), the primary function of the NTE is to "provide objec- 
tive standardized measures of knowledge and skills developed in academic programs for the prep- 
aration of teachers" (p. 3). These measures enable various actors involv3d in teacher training, 
teacher certification, and teacher selection to make useful comparisons with National standards. 
ETS notes that significant information can be obtained from the NTE and argues that the "aca- 
demic preparation for a profession is obviously and substantially related to the future professional 
performance in that profession" (p. 3). 

As noted earlier, most research trying to relate teacher academic abilities, professional prepa- 
ration, and knowledge of subject matter to student outcomes has been inconclusive. Ornstein 
(1984) notes that "at present, we are unable to answer with confidence the influence a teacher has 
on student performance or behavior. The reasons are that the variables are too numerous and the 
interactions and relationships are too complex and multidimensional" (p. 109). 

Setting higher teacher certification requirements may reduce the pool of prospective teachers, 
thereby creating or adding to teacher shortages. The practice may result, among other things, in 
larger teacher-pupil ratios In schools and In more bargaining power for teacher organizations 
negotiating higher wages and better working conditions. 

Some States require vocational teachers to take the NTE to become certified. Although it is a 
viable way of ensuring that those teachers possess basic skills, this screening procedure might 
have the adverse effect of eliminating competent technicians from vocational education. Again, the 
question of test relevancy, validity, and reliability in the area of vocational education must be con- 
sidered fully. 

With regard to test validation. It Is interesting to note that ETS recommends validation of the 
NTE before they are used for specific compliance with Federal civil rights laws relating to disad- 
vantaged and minority groups, such as Title VI and Title VII of the Civil Rights Act of 1964. Further- 
more, ETS (1983) strongly discourages the use of the NTE and their scores for the following 
purposes: 

• Using NTE and their scores as the sole criterion for decisions, whether concerning certifi- 
cation, selection, admission, or program evaluation 

• Using the NTE and their scores directly or indirectly to determine the compensation, 
retention, termination, advancement, pay supplements, or change in provisional employ- 
ment status of teachers once they are employed 

• Using NTE and their scores, whether for certification, selection, or admission, without 
appropriate validation or using qualifying scores without having conducted appropriate 
standard setting studies (p. 10) 

Teacher competency testing, certification, and especiafly periodic recertification can be 
costly. For example, Ishler (1985) indicates that to meet the iesti»^g requirements of Texas House 
Bill 72, an estimated $16.9 million would be required to develop and adrr inister the test to incum- 
bent educators. Is there a real payoff from this kind of investment? Woulc the return be more 
beneficial if the same investment were made on staff development to eumit ste performance defi- 
ciencies and discrepancies? Or, would it not be more beneficial to invest in . ;rengthening teacher 
training institutions? 
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Thd teacher competency testing movement for certification purposes is well on its way and 
gaining greater momentum. Benefits are expected to be considerable, but some of the consequen- 
ces are unknown. Evaluation of the movement's results is necessary. 

Advantages. Following are the nain potential advantages to be derived from teacher compe- 
tency testing: 

• If competency tests are good predictors of teaching effectiveness, then the best teachers 
can be sorted out from poor ones. 

• Competency testing and higher certification requirements may help change the negative 
image of the teaching profession and restore public confidence in education. 

• Competency testing helps ensure that beginning teachers possess the minimum prereq- 
uisite competencies. 

• Competency testing and certification requirements help protect students from exposure 
to incompetent teachers. 

• Competency testing, especially in occupational specialties, may help provide grounds for 
legal defense of vocational instructors implicated in court cases involving student injuries. 

• Since occupational experience is difficult to document accurately, occupational compe- 
tency testing may be a viable way of assessing the wide coverage of content areas, as well 
as the currency of technological knowledge. 

DItadvantagtt. Connelly and de Harta (1985) identify the following additional disadvantages 
of teacher competency testing: 

• There is a negative correlation between NTE scores and achievement of students, espe- 
cially in clinical experiences; there is no research to show a correlation between teachers' 
scores on a written test and their ability to cause pupil achievement. (Pugach and Raths 
1983; Soar. Medley, and Coker 1983) 

• Skills needed in one teaching situation may not be essential in a different situation. For 
example, skills needed for successful teaching in inner-city schools may net be the same 
as those needed in suburban schools. (Sikula 1984) 

• It is unlikely that testing will attract better students to teacher education programs; 
rewards and working conditions are the most important factors in teacher recruitment. 
(Pugach and Raths 1983) 

• Tests tend to discriminate against minorities; testing is likely to reduce minority teachers 
to no more than 5 percent (of the total teacher population! by 1990. (Smith 1984, Gallegos 
1984) 

• Tests emphasize knowledge, not performance, ethical values, creativity, emotional matur- 
ity, or attitudes. (Hyman 1984) 

• Academic quality of teachers is affected by the dynamics of supply and demand in the 
education marketplace, not by testing or more strict certification requirements. (Weaver 
1984) 
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• A certification test favors one teacher education program over another; it can eliminate 
diversity and flexibility in education programs. (Sikula 1984) 

• Tests will narrow educational scope at a time when it needs to be broadened. (Pugach 
and Raths1983) 

• More constraints by the State, such as testing, will result in teacher militancy. (Killian 
1984) 

• it is wasteful and unethical to deny certification on the basis of test scores; if teachers in 
the field are not competent, they should be retrained or dismissed. (Soar. Medley, and 
Coker 1983) 

• Testing is wasting teacher potential in an already small pool (5 percent) of freshmen who 
want to teach. (Gallegos 1984) 

• There are problems of test construction, inasmuch as tests emphasize lower, not higher 
order skills; a single test score is not an adequat3 criterion for decision making. (Thurston 
and House 1981) 

• Reliance on any one criterion (e.g., paper-and-pencil tests) for removing teachers is not 
appropriate. (Sikula 1984. p. 90*91) 

Conditiont for um in vocational education. Today's vocational teachers should have a solid 
academic foundation, be competent in their occupational area, and possess a sound knowledge of 
pedagogy. If prospective vocational teachers are screened at the point of entry into teacher educa- 
tion programs through selective admission, then certification requirements should be drawn so 
that (1) testing efforts are not duplicated and (2) focus is placed on competencies to be acquired 
through the teacher training program. 

In considering the implementation of legislation or regulations for completing testing of voca- 
tional teachers for certification purposes, the following critical conditions should be given ade- 
quate attention: 

• The impact and implications of testing on the supply and demand of candidate teachers 
must be assessed. 

• Whether standardized competency tests are purchased or developed, empirical evidence 
is necessary regarding the soundness, validity, and reliability of these tests. Moreover, 
many test producers delegate the obligation and responsibility of establishing test validity 
and reliability to the test buyers. These are complex activities involving hidden costs that 
must be considered. 

• A cost-benefit analysis of implementing competency testing is desirable to determine if 
the costs irvoived are justified by the potential benefits, or whether investments in staff 
development or other activities for improving teacher quality may be more beneficial. 

• Vocational education has always been faced with the problem of attracting qualified 
individuals from business and industry into teaching because of the Sew profile of the 
teaching profession, higher salaries in the private sector, and the frequent parception of 
teaching as an overworked and underpaid profession. Competency testing might aggra- 
vate these problems and perceptions, further reducing the supply of potential teacher 
candidates. 
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• The full economic, sociological, and political impact of competency testing on minority 
groups must be examined in detail. 

• Trends regarding teacher turnover in vocational education need to be considered to 
determine if the resource commitment to competency testing is justified. 

• Competency testing of the performance component of teachers* occupational compe- 
tency calls for trained examiners, specialized materials, and equipment. Availability and 
cost of these items must be weighed carefully. 

• Testing efforts should not be duplicated. Therefore, legislation and regulations regarding 
the trade competency testing of vocational teachers must take into account individuals 
who have already passed such tests for admission to their professional organizations or 
for obtaining journeyworker certification in their trades. 

• There are many nonquantifiable aspects of human performance that contribute to teach- 
ing effectiveness; these should be part of vocational teacher competency testing efforts. 

Legislators and policymakers need to consider these important conditions in light of their full 
impact and implications before introducing mandatory competency testing for vocational teachers, 
Some responses to these concerns may be readily apparent; others will have to be discovered 
through future research. If competency testing is adopted as a genuine effort to improve voca- 
tional teacher effectiveness, then deeper probing is necessary into the kinds of issues raised here 
(also see figures 1 and 2). 



Conclusion 

Competency testing is one means of reassuring the public of the high quality of teaching and 
instruction. It constitutes part of an important set of tools that are needed to be sure all teachers 
meet minimal qualification requirements for teaching. However, policymakers and State legislators 
must not lose sight of the many limitations of personnel testing. Not all aspects of teaching effec- 
tiveness can be easily quantified. The technology of testing is weak« and consequently, judgments 
based on test results may be less than perfect. Caution in the application of teacher competency 
testing is in order. 
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T'lTLE: 



Teacher Occupational Competency Test (TOCT) 



DEVELOPER: National Occupational Competency Testing Institute (NOCTI) 

DATE: 1973--present 

AVAILABILITY: National Occupational Competency Testing Institute (NOCTI) 

318 Johnson Hall 
Ferris State College 
Big Rapids, Michigan 49307 
Phone (616) 796-4695 

Yes 

Between $180 to $230 per candidate. Cost does no: 'nclude materials and supply 
equipment necessary for performance tests. NOCTI suggests that candidates should 
support testing cost. 

DESCRIPTION: The NOCTI TOCTs are specifically designed for vocational teachers' competencies in terms of the 
theoretical and technical concept of their occupational specialties (cognitive) as well as the performance 
aspect of their trades (psychomotor). The TOCT has two components: (2) written and (2) performance. 
NOCTI recommem': the use of TOCT as a screening device for admissions to teacher education programs. 



Copyright: 
Cost: 



OCCUPATIONAL AREA: Jh , TOCTs are 
Air Conditioning and Refrigeration 
Airframe and Power Plant 
Architectural Drafting 
Auto Body Repair 
Automotive Body and Fender 
Auto Mechanic 
Baking 

Brick Masonry 

Building Construction Occupations 

Building Trades Maintenance 

Cabinet Making and Millwork 

Carpentry 

Civil Technology 

Commercial Art 

Commercial Photography 



available in the following occupation 
Computer Technology 
Cosmetology 
Diesel Enplne Repair 
Drafting Occupations 
Electrical Installation 
Electronic Communications 
Electronic Technology 
Heating 

Industrial Electrician 
Machine Drafting 
Machine Trades 
Major Appliance Repair 
Masonry 

Masonry Occupations 
Materials Handling 



al specialties: 
Mechanical Technology 
r'ainting and Decorating 
Plumbing 
Power Sewing 
Printing-Letterp'ess 
Printing-Offset 
Quantity Food Preparation 
Quantity Foods 
Radio/TV Repair 
Refrigeration 
Sheet Metal 

Textile Production/Fabrication 
Tool and Die Making 
Welding 



COMING SOON: 

Building and Home Maintenance Services, Heavy Equipment Mechanic, and Law Enforcement 



Number of Itamt: 



Typt of itamt: 

(Written) 

Intandad Population: 



ADMINISTRATION: 
Whan: 

How: 

Time Racommandad: 

Scoring: 

TEST DATA: 
Reliability: 

Validity: 



Between 100 and 200 
Multiple choice 

Prospective vocational teachers 
Selective admission or certification 

On*site, written test-paper-and*pencil; and performance test-work sample 

Varies according to occupational specialties. In general, written tests require 3 hours 
to administer and the performance tests, between 4 and 6 hours. 

Performance tests are scored on-site by appointed ex£miner. and written tests are 
sent to NOCTI central office for scoring. 

All tests developed are extensively field-tested nationaily to establish reliability usmg 
the KR20 and the Cronbach ALPHA. 

All tests are validated in the field with the help of incumbent workers. 



Figure 1. The Teacher Occupational Compentency Test (TOCT) 
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TITI C. 

TITLc: 


Vocational Teacher Competency Testing 


DEVELOPER: 


National Evaluation Systems, Inc. (NES) 


DATE: 


1972-1985 


AVAILABILITY: 




For Further 
Information: 


Michael L.Chernoff, Ph.D. 
National Evaluation Systems, Inc. 
301 Gatehouse Road 
Amherst, Massachusetts 01022 
Phone (413) 246-0444 


Copyright: 


Yes 


Cost: 


Contact the National Evaluation Systems, Inc., for more information 


DESCRIPTION: These tests are all criterion-referenced objective-based tests of knowledge and skills in the related 
occupational specialties. They were designed for and are used in teacher certification testing programs. 


OCCUPATIONAL AREA: 


The NES tests are available in the following occupational specialties: 

Accounting Consumer Finance 
Agriculture Distributive Education 
Audiovisual Specialist Drafting 
Basic Skills Home Economics 
Business Economics Industrial Arts 
Business Education Marketing 
Business English Metal Work 
Business Law Office Procedures 
Business Machines Shorthand 
Business Mathematics Typewriting 
Career/Occupational Woodwork 
information 


Number of Items: 


Operational forms of the tests include 100-120 scorable items from a larger pool of 
Items. 


Type of Items: 


All items are multiple choice. 


Intended Population: 


These tests are designed for individuals seeking initial certification in the related areas. 
They can also be used for other certification-related purposes. 


Time to Administer: 


2-3 hours. 


Response Mode; 


Paper and pencil (optically scanned answer sheets). 


Scoring: 


ALL scoring and reporting services are provided by the National Evaluation Systems, 
Inc. 


TEST DATA: Extensive statistical information exists on test materials from field tests and operational adminis- 
trations. Please contact the National Evaluation Systems, Inc., for more information. 



Figure 2. The National Evaluation Systems' Vocational Teacher Competency Testing 
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TMcher Interviews 



Teacher interviews are one approach available to administrators of educational institutions for 
evaluating instructional personnel. In spite of some apparent limitations, the interview process can 
yield information about teachers that no other methods can generate. This discussion will focus on 
the two most basic applications of teacher interviews: teacher selection and teacher appraisal. 



Teacher Selection Interviews 

Admission and hiring interviews can be used to assess the interpersonal skills of candidates, 
provide information not easily available from education records and application forms, and give 
the interviewer an opportunity to judge many personal characteristics of applicants. This oppor- 
tunity is reciprocal, because candidates can also ask questions, make judgments, and clarify per- 
ceptions and concerns. 

Feistritzer (1984) reports that 52 percent of the teacher training institutions surveyed by the 
National Center for Educational information (NCEi) indicated that they used interviews as a tool to 
screen applicants for admission to teacher education programs. Selection interviews can be struc- 
tured or unstructured. Structured interviews with clearly defined objectives usually generate more 
reliable and valid information about job applicants. Selection interviews can also be conducted by 
one interviewer, or they can be administered by a group of interviewers to bring extra expertise 
into the selection process. 

Issues in use. In selection interviews, the emphasis is primarily on teacher characteristics, pro- 
fessional non-classroom-related activities, and other personal nonprofessional activities and inter- 
ests. These are all indirect, high-inference measures of teaching competence. As King (1981) 
notes, **there is no single set of skills, attitudes, interests, and abilities that all good teachers have 
and that all poor teachers lack" (p. 174). This fact seriously affects the validity and reliability of 
selection interviews. 

Advantages and disadvantages. Teacher selection interviews have various advantages and 
disadvantages that must be taken into account in any decision to use them as a screening proce- 
dure. Among other things, they provide the following: 

• A unique opportunity for face-to-face interaction between interviewers and interviewees 

• An indication of the verbal communication skills of candidates under stressful conditions 

• An opportunity to assess candidates on other dimensions of competence 

• An opportunity for the interviewee to learn more about the job, intended responsibilities, 
obligations, and reward system 

Following are two major disadvantages of teacher selection interviews: 

• Information gathered by selection interviews usually is colored with interviewer biases. 

• Teacher selection interviews are high-inference, subjective screening techniques that, to 
be effective, must be used in conjunction with other selection techniques and information 
that can provide valid aiid reliable information. 
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Conditions for um In vocational tducatlon. In spite of its limitations, the selection interview, 
when used in conjunction with other sources of data, can help administrators make more informed 
decisions. A team of interviewers is sometimes used in selection interviews. Such a practice is par- 
ticularly useful In vocational education, where school district administrators or principals may not 
have the technical background and expertise necessary for adequately assessing the technical 
competency and experiences of prospective vocational teachers. The interview process can be 
strengthened by assembling a selection committee made up of administrators competent in per- 
sonnel matters, educators with a solid background in teaching and learning theories, and voca- 
tional educators with subject-matter expertise in the specific occupational area. Group or team 
interviewing also helps to eliminate personal bias on the part of the interviewers. 

Good interviewing skills can be developed and are essential prerequisites for conducting suc- 
cessful interviews. To be successful, interviewers must be skilled in the following areas: listening; 
questioning; and processing, analyzing, interpreting, and evaluating information obtained through 
interviews. 

When properly used, the selection instruments can be imoortant tools. Most instruments avail- 
able forteacher selection interviews are primarily designed for selecting academic teachers. 
Therefore, care must be exercised in choosing interview instruments to be sure ihey adequately 
reflect any special needs of vocational education. 



Teacher Appraisal Interviews 

When used as a teacher evaluation strategy, the teacher appraisal interview can have multiple 
applications and objectives. Haefele (1961) identifies the following specific applications: 

• Warning that improvement in teaching is needed 

• Establishing teaching performance standards 

• Motivating teaching performance 

• Assessing the degree to which teaching performance standards have been achieved 

• Deterniining what the evaluator or others can do to sustain or improve teaching perfor- 
mances (p. 51) 

Darling-Hammond, Wise, and Pease (1983) indicate that although classroom observation fol- 
lowed by an appraisal interview with the teacher represented the totality of the teacher evaluation 
process in the past, there has recently been a major shift in the use of interviews for teacher eval- 
uation. They note that 

recently ... the interview has been viewed as an important element of a broader evalua- 
tion procedure. In particular, a preobservation conference has been recognized as useful 
for the involvement of teachers in their own professional development and for a more 
regularized exchange of feedback from supervisors, (p. 305) 

The teacher appraisal interview is not a stand-alone approach. However, when used with other 
approaches, it can enhance the teacher evaluation process. When administered prior to the evalua- 
tion, an interview gives teachers and evaluators an opportunity to set goals and standards and to 
agree on the evaluation focus. When given after the evaluation, the appraisal interview brings 
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teachers and evaluators together to discuss the evaluation results, clarify any ambiguity, and set a 
plan of action for improvement. Standardized instruments and interview guides can be used to 
increase the objectivity of the information gathered through the performance interview procedure. 
Teacher appraisal inierviews can be used as effectively in vocational education as in general 
education. 

IMUM In use. Teacher appraisal interviews can be used for both improvement and personnel 
decisions. When used for improvement, the interview can be conducted by peers, supervisors, or 
staff development personnel. However, when used for personnel decisions, such as salary and 
promotion decisions, the credibility of the process becomes an issue and the interview should be 
conducted by supervisors. 

Appraisal interviews can be the source of in*depth qualitative data that are extremely useful 
for focusing evaluation activities and improvement efforts. However, because of the two potentially 
conflicting purposes of the teacher appraisal interview— formative and summative evaluation- 
prior agreement must be reached regarding the explicit u&e of the Information generated for each 
purpose to protect teachers from misuse of inappropriate information. 

Advantagts and dlMdvantagts. The major advantage of using the teacher appraisal interview 
is that it provides an excellent opportunity for briefing sessions to focus evaluation efforts, as well 
as for debriefing sessions after the evaluation to make an action plan for improvement, it also pro* 
vides rich qualitative data that are valuable for formative evaluations and that can only be obtained 
through such interviews. One disadvantage is that when the appraisal interview is used for summa- 
tive purposes, the evaluator*s subjectivity may seriously affect the validity and reliability of the 
information obtained. 

Conditions for um In vocational education. To increase the validity of the information 
gathered through the performance appraisal interview, the interviewer should be knowledgeable 
about theories of instruction generally and vocational education instruction specifically. In some 
instances, when evaluating critical competencies related to occupational specialties, the inter- 
viewer may also need to have an in*depth knowledge of the subject matter. 

Teacher appraisal interviews also have a great potential when used in conjunction with other 
teacher evaluation strategies. At the initial stage, they can be used for setting the focus and 
parameters of the evaluation. At the postevaluation stage, the interview process can be used to 
connmunicate results and to draw up an action plan for making improvement. The potential of the 
appraisal interview is particularly applicable in assessing vocational teachers, where evaluation of 
some competencies may call for advance planning to assemble specialized equipment and mate- 
rials. The preobservation appraisal interview can help clarify and identify these needs. 
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student Achievement 



The issue of evaluating teachers based on their students' achievement has always generated 
heated debates. Because of the controversy surrounding this teacher evaluation strategy, it has 
never gained general acceptance for full-scale implementation. In this period of educational 
reform, the teaching profession is under close scrutiny, and there is an ever-increasing call for bet- 
ter and more teacher evaluation. Educators are likely to encounter requests to use student 
achievement as a teacher evaluation strategy. It is therefore examined next to help educators think 
about its strengths and weaknesses. 



Overview 

The growing concern for accountability in public education has led some educational adminis- 
trators, policymakers, and researchers to view education from the perspective of the industrial 
input-output model. In this model, the teacher functions as the input and, through the process of 
instruction, generates the product, namely, the students* gain in learning. The process of instruc- 
tion viewed in this narrow sense makes teachers directly responsible for students' learning, as 
measured by scores on standardized tests. This proposition is interesting from a political viewpoint 
for two reasons. First, it is based on an accountability system that can be easily monitored. 
Second, it places all the obligation for student progress directly on teachers. It should be noted, 
however, that there is no body of research in vocational education to indicate a relationship 
between teaching effectiveness and student outcomes. Interestingly, in spite of this lack of empiri- 
cal evidence, conventional wisdom within teacher education has traditionally held that if the stu- 
dent has not learned, the teacher has not taught. 

There are three dimensions to teacher competency, namely, teacher competence, teacher per- 
formance, and teacher effectiveness (Medley, Coker, and Soar 1984). Teacher competence is 
viewed as a presage variable, that is, the knowledge, skills, and attitudes that a teacher possesses. 
Teacher performance is related to the process of teaching, and teacher effectiveness is related to 
the capabilities of the teacher to successfully impart knowledge to the students. Teaching effec- 
tiveness is one of the most difficult dimensions of a teacher's competency to evaluate. It has been 
argued that if the entry behaviors of students before instruction and their exit behaviors after 
instruction are known, learning gain, and thus teacher effectiveness, car be established. However, 
many methodological and measurement problems plague such estimates. 

ivicNeil and Popham (1977) identify the contract plan as an alternative method of evaluating 
toachers' effectiveness using data on students' learning gain. In setting up a contract plan, the 
teaciier and the administrator develop a set of student objectives and a set of criteria for measur- 
ing the attainment of these objectives as a result of instruction. The contract period for which the 
teacher is accountable can vary from a single day's lesson to a semester or a whole year. 

McNeil and Popham (1977) suggest the use of performance tests as a valid and fair proposi- 
tion for measuring teacher competency. They describe a performance test as a test of teaching 
power, in which several teachers are given a set of identical objectives, usually novel to both 
teachers and students, and are asked to plan and teach a lesson that accomplishes these objec- 
tives. In this situation, the students are randomly assigned to teachers who use their own judgment 
in the selection of appropriate instructional strategies. At the end of the lesson, a test is adminis- 
tered by a third party to assess the students' achievement of the identified objectives, and the 
mean test score is used as a criterion of effectiveness. McNeil and Popham indicate that a number 
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of studies have shown that some teachers '*are consistently more successful than others in getting 
results" (p. 273). They suggest that there is a need to verify the finding that 



teachers who can produce desired effects under conditions of teaching performance 
tests maintain their effect over time and in the presence of a greater range of conditions 
such as effective use of their instructional time and in targeting tneir instruction to areas 
in which learning is most desired, (p. 147) 

These varfous propositions for evaluating teaching effectiveness using student achievement 
seem attractive enough at first glance. However, their i.nplementation is plagued by several con- 
founding issues, some of which are discussed next. 



lituts in Use 

Two fundamental types of problems— philosophical and methodological— surface in any 
attempt to evaluate teaching based on measures of student learning (Doyle 1983). Doyle argues 
that **it seems unfair to evaluate a person in terms of what other people do or fail to do, at least 
when those other people are at best only partially under the control of the person being evaluated" 
(p. 10). Such a situation can be very counterproductive to teacher growth and student develop* 
ment and can only, in the long run, perpetuate mediocrity in our schools. 

Another problem advanced by Doyle (1983) against making teachers accountable for students' 
achievement is methodological. He argues that 

the technology of testing and statistical analysis is not yet advanced enough for accept- 
able and sure conclusions of this sort. First, a testing procedure for evaluating teachers 
would need to account, at least, for student ability, student knowledge at the beginning 
of the course, and student motivation throughout the course, so that an instructor does 
not appear good or poor simply because of the capacities of the students in the class, 
(p. 10) 

Glass (1974) asserts that it is "invalid and unfair" to evaluate teaching effectiveness on the 
basis of students' achievement on standardized tests. He further argues tnat "standardized tests 
will uncover gross educational deficiencies in basic skills, but such instruments are not designed 
to reveal the variety of ways in which teaching and learning can be creative, favorable, opportunis- 
tic, and uniquely meaningful to students" (p. 11). 

Glass (1974) also states that all attempts to evaluate teachers on the basis of student standard- 
ized test results are unjust because much of a standardized test's content is irrelevant to teaching 
situations and because it is not possible to account for variations among individual students' entry- 
level abilities. According to Glass, statistical manipulation of students' scores cannot overcome 
this problem, and only random assignment of students to teachers could ensure fairness in the 
process. Glass also argues that "any attempt to measure teacher effectiveness through standard- 
ized test scores of students would inevitably result in a tendency for teachers to teach for the 
test— teach the 'safe' topics— possibly at the expense of the more elusive and more important 
ones" (p. 12). 

Educators have always been suspicious of providing instruction directly correlated to tests 
measuring s* Jdents' achievement. However, an examination of most models for instructional 
development show that test items are usually written before instructional activities are determined 
and that these test items are then criterion-referenced to the instructional objectives. \i teaching 
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for the test can improve students* achievement of course objectives, then this approach has more 
merit than teaching without any specific target. In vocational education, where many delivery pro- 
grams are competency based and where test items are referenced to enabling objectives, helping 
students to attain mastery of these objectives should also help them score high on achievement 
tests, provided these tests are also criterion referenced. 

When teaching is viewed from the perspective of a process-product model, the only way to 
evaluate "products** is to look at the amount of learning gained by the students as a result of 
instruction. Borich and Fenton (1977) indicate that "process measures assess teacher perfor- 
mance, while product measures assess pupil change. Though pupil outcomes ca«i be unstable and 
difficult to measure, they nevertheless must stand as one index of teacher effectiveness" (p. AH), 

Formative evaluation. In the midst of this controversy, McGreal (1983) suggests that, "to main- 
tain emphasis on instructional improvement that characterizes effective evaluation systems, an 
informal approach to monitoring and using student performance data seems most useful" (p. 130). 
Millman (1981 ) identifies a set of criteria for generating good student achievement measures for 
the formative evaluation of teachers. He feels that achievement measures should be "free from 
teache ' bias," both in their construction and in their scoring. The measure should have a "diagnos- 
tic function indicating students* greatest strengths and weaknesses so that teachers are able to 
modify instruction most effectively" (p. 148). The test items should be reliable and broadly con- 
ceived to assess both intended and unanticipated learning outcomes. 

Millman (1981) also provides some useful guidelines regarding methods of data collection on 
student achievement for teacher evaluation purposes. He suggests the use of baseline data from 
curriculum-embedded tasks, such as completed homework, in-class assignments, answers to 
questions posed by the teacher, questions asked of the teacher, and students* self-appraisal 
(p. 150). 

Summative evaluation. Summative evaluation usually is more threatening, more controversial, 
and deeper in the implications than is formative evaluation. Consequently, attempts to measure 
teaching effectiveness for personnel decisions often result in heated debates. Whereas some indi- 
viduals view stuc'ent achievement measures as the only relevant source of data for assessing 
teaching effectiveness, others argue that "too many confounding factors affect student learning 
besides the teacher*s performance" (Millman 1981, p. 156). 

Research studies indicate that teachers cannot be held totally accountable for student out- 
comes. Bloom (1982). in his classic work entitled Human Characteristics and School Learning, 
notes that 

cognitive entry behaviors can account for about 50 percent of the variation in achieve- 
ment, while affective entry characteristics alone account for about 25 percent ... in 
combination these entry characteristics could account for about 60 percent of the 
achievement variation on a new set of learning tasks, (p. 108) 

He further adds that 

we doubt that the quality of instruction can overcome the effect of lack of the prerequi- 
site cognitive entry behaviors The lack of the necessary prerequisite cognitive entry 

behaviors for a particular learning task should make it impossible for the student to mas- 
ter the learning task requirements no matter how good the quality of instruction for that 
task. (p. 108) 
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Ornstsin (1984), commenting on research findings of teacher effect on student outcomes, 
points out that 



until replicable findings of teacher effectiveness can be established, the notion of 
teacher evaluation, teacher accountability, teacher performance and teacher competen- 
cies are nonworkable. If such empirical data are lacking, then various forms of teacher 
effectiveness have been sold before they exist in reality, (p. 109) 

Some researchers have tried to establish the relationship between teaching and student 
achievement. The Rosenshineand Furst (1971) review of teacher process-product studies showed 

teacher process strongly and consistently related to student achievement or attitude. 
The first five [teacher processes] were the strongest: clarity, variability, enthusiasm, task 
orientation, and student opportunity to learn. The next six were characterized as promis- 
ing: use of student ideas, justified criticism, structuring comments, various types of 
questions, probing and encouraging student elaboration, and high standards of perfor- 
mance, (p. 112) 

Gage's (1978) analysis of 49 process-product studies showed 4 types of teacher behaviors hav- 
ing strong relationships with student outcomes. These behaviors were teacher indirectness praise, 
acceptance, and criticism. After conducting a comprehensive and sophisticated review of 289 
teacher process-product studies. Medley, Coker, and Soar (1984) concluded that "effective 
teachers behave differently with different types of students" (p. 113). 

The Beginning Teacher Evaluation Study (BTES) has provided a consistent, important finding 
about teacher effectiveness— that academic learning *«me is the most important variable associated 
with student learning. Academic learning time is defined as the amount of time a student spends 
attending to academic tasks while also performing with a high success rate. In other words, both 
the quality and the quantity of learning time on task are considered important in improving student 
learning. 

There seems to be little or no agreement among researchers as to the influence of teachers on 
student learning outcomes. Whereas many support the thesis that teachers do have a significant 
influence on student achievement, others conclude that teachers* influence on student learning is 
null or negligible. In considering the use of student achievement measures to evaluate teaching 
effectiveness, it important to distinguish between student achievement of trivial, easy learning 
objectives and achievement of important and difficult educational objectives. A teacher who 
helped students consistently achieve more important learning objectives snould receive a higher 
rating of effectiveness than one whose students were achieving mostly easier learning objectives. 
This kind of analysis is important and useful for providing greater insight into student achievement 
not evidenced by test scores alone. 

The uncertainties surrounding the relationship between teaching effectiveness and student 
achievement suggest that student outcome data should not be used in teacher evaluations for 
summative purposes. Until more is known about the relationship between toaching effectiveness 
and student outcomes, teachers should be encouraged to use student learning data for self- 
improvement purposes only. 




44 



52 



AdvantagM and DiMdvantagts 



When used as a formative, self-evaluation strategy, student achievement measures can help 
teachers identify weaknesses and seek help in improving their teaching. However, there are several 
major disadvantages to using student achievement measures as an indication of teaching effec- 
tiveness. Some of these are as follows: 

• The relationships between teacher effectiveness and student outcomes have not been 
reliably established. The stability of teacher influences is low. 

• Too nfiany factors beyond the control of teachers contribute to learning gains among stu- 
dents, such as prerequisite cognitive entry behaviors and affective entry behaviors. 

• Gain scores, not merely end-of-instruction achievement measures, are needed to estimate 
achievement, but such measures are hard to obtain and difficult to assess. 

• Testing technology is too weak to generate valid and reliable standardized measures of 
student gain. 



CondiUont for Use in Vocational Education 

The limits identified regarding the use of student achievement data for teacher evaluations cut 
across subjects and grade levels of education. Consequently, the use of such data as the basis for 
sunnmative judgments about vocational teachers' effectiveness is inappropriate. Its use in personal 
evaluations for seh improvement purposes has some merit. 

Instruction in vocational education is aimed at developing student skills, knowledge, and atti- 
tudes. Attempts to assess teaching effectiveness on the basis of student achievement must take 
these three components into account inasmuch as they are all critical elements in the total prepa- 
ration of vocational education students for the workplace. Additionally, vocational teachers are 
informally assessed on the basis of tangible student achievement indicators, such as completed 
projects, accident reports, employer evaluations of students, and employers' satisfaction with 
graduates. 

Student learning gains related to skills, knowledge, and attitudes must be assessed indepen- 
dently. Bloom's (1982) taxonomy of educational objectives and related learning domains may be 
useful in developing tests. Often, tests designed to gauge student achievement are written to 
reflect behavior found at the lower end of the domains of learning. This is largely because test 
items for measuring complex learning at higher levels are more difificult to write. 

In general education, student achievement is ..sually measured on the basis of paper-and- 
pencil tests. Vocational education offers an additional dimension to evaluation, through the correct 
demonstration of acquired competencies. At least two major types of National standardized tests 
are available for assessing student achievement in vocational education: (1) tne Student Occupa- 
tional Competency Achievement Tests (SOCATs) developed by NOCTI and (2) the Vocational 
Competency Measures (VCMs) developed by the American institutes for Research (AIR). The 
SOCATs are available in 32 different occupational areas, and the VCMs cover 17 different occupa- 
tions. Both tests contain cognitive and psychomotor assessment components and are typically 
used as terminal measures for summative evaluation, that is, for generating end-of-training data 
related to overall student achievement. The validity and reliability of the SOCATs and VCMs have 
been established through extensive and carefully designed National field trials and revisions. 
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NOCTi has identified the following benefits that may be derived from use of the SOCAT. 

• Students— Secondary and postsecondiry students will be able to demonstrate their com- 
petence on a National standardized test and use the test results as a credential for 
employment and advar^ced standing in educational programs. 

• Teachers^Teachers, by examining the strengths and weaknesses of their students, will 
be able to improve their curriculum for maximum student achievement. SOCATs can not 
only provide an objective means of grading, but they can be used to help document 
teacher effectiveness. 

• Vocational administrators— SOCAT results reported for classes, schools, and States will 
enable vocational administrators to assess relative strengths and weaknesses of programs 
more acci lately and will indicate whether present curriculums are up to date and valid 
relative to occupational requirements 

a Employers— SOCAT results will enable empfoyers to assess and hire individuals on the 
basis of demonstrated competence, a^ opposed to mere letter grades and course titles. 

The profiles at the end of this section provide more detailed information ^borit the SOCAT and 
VCM (see figures 3 and 4). 

in spite Oi ail the shortcomings associated with the use of student achievement measures for 
evaluating teacher effectiveness, there is growing evidence that the public quest for quality and 
accountability in education will encourage greater reliance on student data as a primary indicator 
of teacher effectiveness. As advancements in instructional and assessment technology make it 
possible to monitor student progress more closely, perhaps more reliable and defensible measures 
and techniques will be developed for evaluating teacher effectiveness in terms of student learning 
gains. Until such time, student achievement measures still have a legitimate and useful place in 
self-initiated and self-administered teacher evaluation efforts geared toward self-improvement. 
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TITLE: 


Student Occupational Competency Achievement Test (SOCAT) 


DEVELOPER: 


National Occupational Competency Testing Institute (NOCTI) 


DATE: 


1973-present 


AVAILABILITY: 


National Occupational Testing Institute (NOCTI) 

318 Johnson Hall 

Ferris State College 

Big Rapids. Michigan 49307 

Phone (616) 796-4645 


Copyright: 


Yes 


Cost: 


Varies according to quantity purchased and membership of purchase. 

SOCAT PR ICE SCHEDULE 
(effective July 1. 1984) 


Quantity Ord^rtd 

10-100 
101 200 
201-300 
301-up 


Full Mambar Atsoclata Mambar Nonmambar 

$6.00 $6.75 $7.50 
5.50 6.25 7.00 
5.00 5.75 6.50 
4.50 5.25 8.00 


DESCRIPTION: The SOCAT is designed to assess student end-of-training competencies in thier area. All SOCATs 
are written tests designed to assess learning in the cognitive domain. 


OCCUPATIONAL AREA: 


The SOCAT is availabe in the following occupational specialties: 

Accounting/Bookkeeping Horticulture 
Agriculture-Mechanics Industrial Electricity 
Auto Body Industrial Electronics 
Auto Mechanics Machine Trades 
Carpentry Plumbing 
Commercial Foods Practical Nursing 
Computer Programming Printing 
Construction Electricity Radio and TV Repair 
Drafting Sewn Products 
General Merchandising Smalt Engine Repair 
General Office Welding 
Heating and Air Conditioning 


Numbar of Itams: 


Varies with each test 


Typa of Itams: 


Multiple choice 


Intandad Population: 


Graduating seniors in vocational education. 


ADMINISYRAFION: 




TIma: 


Varies with each test. 


Rtsponte Moda: 


Paper and pencil. 


Scoring: 


All tests are scored by NOCTI. 


TEST DATA: 




RaliabilityA/alidity: 


NQ3TI field-tests all SOCAT Instruments for validity and reliability. 



Figure 3. The Student Ocojpational Competency Achievement Test (SOCAT) 
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TITLE: 
DEVELOPER: 

DATE: 

AVAILABILITY: 



Ohio Vocational Education Achievement Test Program (OVEAT) 

Vocational Institutional Material Laboratory 
The Onio State University 



The OVEAT is available in the following occupational speciality areas: 



!964-pre$ent 

Vocational Instructional Materials Laboratory 
The Ohio State University 
1885 Neil Avenue 
Columbus, Ohio 43210 

Copyright: Yes 

Co«t: Ohio schools: March Order-$1.30/student 

Out-of^ate schools: March Order-$2.50/student 
All schools other than March Orders-$3.00/student 
Cost includes test instruments, scoring, and reporting 

DESCRIPTION: These tests are specifically designed for use by teachers, supervisors, and administrators for evaluation 
and diagnosis of vocational achievement for the improvement of instruction. The test battery consists of three 
components: (1) The California Short Form Test of Academic Aptitude (SFTAA); (2) Vocational Achievement 
Test, Part 1; (3) Vocational Achievement Test, Part 2. The test battery is administered on 3 consecutive days. 

OCCUPATIONAL AREA: 
Agricultural Education 

1. Agricultural Business 

2. Agricultural Mechanics 

3. Farm Management 

4. Horticulture 

5. Production Agriculture 
Health Occupations Education 

1. Dental Assisting 

2. Medical Assisting 

3. Diversified Health 
Occupation 

Construction Trades 

1. Carpentry 

2. Construction Electricity 

3. Heating, Air Conditioning, 
and Refrigeration 

4. Masonry 

Metal Trades 

1. Machine Trades 

2. Welding 



Businass and Off ica Education 

1. Accounting/Computer Clerk 

2. Cleric Stenographer 

3. Cleric Typist 

4. Data Processing 

5. General Office Clerk 
Home Economics Education 

1. Community and Home Service: 

2. Fabric Services 

3. Food Services 

4. Nursery School Teacher Aide 
Electronics 

1. Communication Products 
Electronics 

2. Industrial Electronics 



Personal Services 

1. Cosmetology 



Distributive Education 

1 . Apparel and Accessories 

2. Food Marketing 

3. Food Service Personnel 

4. General Merchandising 

Trade and Industrial Education 

1. Auto 8ody Mechanics 

2. Automotive Mechanics 

3. Diesel Mechanic 

Graphic Communications 

1. Commercial Art 

2. Drafting 

3. Lithographic Printing 



Equipment 

1. Small Engine Repair 



Number of Itams: 
Typas of Itams: 
Intended Population: 
Respondent: 

ADMINISTRATION: 
Whan: 
How: 

Time Recommended: 
Day 

1 

2 
3 

Response Mode: 
Scoring: 
TEST DATA: 
Reliabifity: 

Validity: 



Between 150 and 200. 

Multiple choice, with four pos^ .le responses. 

Secondary vocational education students in grades 1 1 and 1 2. 

Student. 

Annually, preferably in March 
On*site 



Test 
SFTAA 

Vocational Achievement Test, Part 1 
Vocational Achievement Test, Part 2 



Time 

1 hour 

2 to hours 
2 to hours 



Separate answer booklet 

Scoring and reporting services are provided by the test developers. 

^^20*21 above 
Content and Construct 



Figure 4. The Ohio Vocations! Fducation Achievement Test Program (OVEAT) 
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Classroom Observation 



Introduction 

Classroom observation is perhaps one of the earliest developed approaches used for evaluat- 
ing teachers and is still the most universally used teacher assessment strategy. It is "the mainstay 
of most teacher evaluations** (Darling-Hammond, Wise, and Pease 1983, p. 306). 

Teacher performance can best be assessed by documenting the teaching process, and class- 
room observation provides a unique opportunity for a third party to obtain '*a view of the climate, 
rapport, interaction, and functioning of the classroom available from no other source" (Evertson 
and Holley 1981, p. 90). As its name implies, classroom observation consists of observation of the 
teaching act by an observer who can be an administrator, supervisor, or peer, which results in a 
judgment about the teacher's performance based on some predetermined set of criteria. The pri- 
mary emphasis is on the process of teaching. Some kind of instrument is generally used to focus 
the assessment and to ensure greater objectivity by observers. 

Two basic approaches are currently used for evaluating teachers through classroom observa- 
tion: (1) classroom visits and (2) systematic observations. Both of these approaches are described 
next, followed by profiles of instruments appropriate for these approaches. 



Classroom Visits 

Description. Teacher evaluation conducted through classroom visits employs short-term, 
data-gathering visits by administrators, supervisors, or peers (Peterson and Kauchak 1982). It is 
the most widely used teacher evaluation strategy, primarily because of its cost-effectiveness and 
ease of administration. Checklists, rating scales, and narrative reports are typically used in con- 
junction with classroom visits for data-recording purposes. Sometimes, videotaping is used to 
record teaching behavior and classroom activities. 

In most instances, classroom visits are short, usually once-a-year teacher appraisal efforts. 
Consequently, not much credibility is ordinarily attached to data gathered in this way, and few crit- 
ical personnel decisions are. made solely on the basis of this evaluation strategy. 

Issuas In use. Although the classroom visit appears useful for evaluating teachers, the strategy 
is typically plagued with problems that affect the validity and reliability of the results. Such prob- 
lems i^^clude observer bias, insufficient sampling of performance, and poor measurement 
instruments. 

In explaining how observer bias affecis information gathered through classroom visits. 
Peterson and Kauchak (1982) note that "due to biases, lack of perception of all that occurs, and a 
limited personal perspective, much that is relevant is missed and that which is noted fails within a 
personal frame of reference** (p. 20). In current practice, teachers often are evaluated by infrequent 
classroom visits— in some instances as seldom a once a year. This "snapshot** approach to the eval- 
uation of teacher classroom performance cannot provide defensible information for personnel 
decisions or meaningful guidelines for improvement of teaching effectiveness. Peterson and 
Kauchak (1982) argue that "classrooms are very complicated places, and they change over time. 
For th3se reasons many visits are required to observe teachers in a representative manner** (p. 21). 

Data collection is a major problem jeopardizing the validity and reliability of classroom visits. 
Three data-gathering strategies are commonly used in conjunction with classroom visits; 
(1) checklists. (2) rating scales, and (3) narrative reports. 
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Usually, a checklist or a rating scale identifies predetermined, desirable teacher behaviors. 
When using a checklist the observer simply checks the occurrence or frequency of a behavior. A 
rating scale is used not only to record the occurrence of a behavior but also to make judgments 
about the quality of a teacher's performance. 

The narrative report is an alternative technique that can be used for data collection during 
classroom visits. Observers attempt to report completely and objectively the activities taking place 
during classroom visits, as well as their sequence. The process can be enhanced by audiotaping or 
videotaping to provide more objectivity and a more in-depth description of the teaching process. 
Each of these data-recording techniques represents a high-inference measure of teaching 
effectiveness— a fact that raises serious questions about their objectivity, validity, and reliability. 

Peterson and Kauchak (1982) identify some of the following problems associated with check- 
lists and rating scales: 

• Checklists and rating scales often contain items that are difficult to observe or are not 
related to student learning. 

• Rating forms often have both formative and summative uses, which interferes with the 
function of either intention. 

• The test forms contain too many items. 

• The great majority of the rating forms in current use have not been checked for reliability. 

• Most forms have not been developed on the basis of empirical foundations related to 
teaching effectiveness. 

• These forms often fail to facilitate the summary of the evaluation effort, (p.. 21) 

In spite of their inherent attractiveness, classroom visits have many limitations that reduce 
their utility and effectiveness as a teacher assessment strategy. After reviewing numerous studies 
on classroom visits, Evertson and Holley (1981) conclude that "there is a fairly consistent failure 
between ratings of teacher performance and other external measures of competence** (p. 96). 
However, according to Peterson and Kauchak (1982), "the question of the soundness of classroom 
visits as an evaluation technique is overlooked because of their widespread use" (p. 21). As pres- 
ently practiced, classroom visits are not an effective means for accurately diagnosing teacher 
strengths and weaknesses. Scriven (1981 ) observes that to evaluate teachers through classroom 
visits "is not just incorrect, it is a disgrace" (p. 251).. 

Advantages and diMdvantaget. Like other teacher assessment strategies, classroom visits 
have several pros and cons associated v/ith them. 

On the positive side, they— 

• are a cost-effective and timesaving way for busy administrators to meet local require- 
ments for teacher evaluation; 

• enable a rater to note any gross deviation from accepted norms for teaching practices; 
and 

• provide the raters with opportunities to evaluate classroom organization, appearance, and 
neatness— conditions that cannot be assessed in any other way. 
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On the negative side, classroom visits— 



• lack objectivity, validity, and reliability; 

• generate high-inference measures that often are not defensible and that have limited 
application in personnel decisions: 

• generally do not provide a representative sample of the wide repertoire of teacher compe- 
tence, especially if they are short and infrequent; and 

• are often disruptive of normal classroom routine and behavior and can be humiliating for 
the teachers. 

Additionally. Scriven (1981 ) identifies the following disadvantages related to classroom visits: 
(1) they alter the classroom and consequently alter teaching. (2) they do not provide a representa- 
tive sample of teacher behavior because they are usually too few in number. (3) independent per- 
sonal prejudices can creep into the evaluation and the evaluator's judgment for or against the 
teacher. (4) information gathered during visits does not readily permit the generation of valid 
inferences as to the quality of teaching, and (5) the influence of the evaluator's personal bias is too 
strong in this approach. 

Applications in voc^tionai education. In vocational education, the laboratory visit or site visit is 
frequently used to evaluate the management and organizational skills of vocational teachers. Such 
skills are frequently assessed with respect to the following areas: workshop organization and man- 
agement: workshop procedures: safety standards; procedures for the care and maintenance of 
equipment, tools, and machines: housekeeping practices: inventory control; budgeting and cost 
control: and student project storage and control. These are all important aspects of a vocational 
teachers daily responsibilities that contribute to efficient and safe instruction. Nevertheless, 
because they are not teaching behaviors per se. their assessnr>ent may reveal little about teaching 
effectiveness or learning achievement. In fact, the relationship of such classroom or shop man- 
agement skills to effective teaching is uncertain. 



Systtmatic Observation 

The systematic observation approach to teacher evaluation consists essentially of observing a 
classroom teacher and documenting and analyzing classroom performance. Peterson and 
Kauchak (1982) note, ''classroom observation is systematic when it fairly represents what goes on 
in classrooms: can be agreed upon by knowledgeable persons: and when the content of the obser- 
vation is defensible in terms of its educational importance** (p. 34). Lyons. Morris, and Fitz-Gibbon 
(1978) state that observation is systematic **when the observer is told what to look at beforehand** 
(p. 90). When used properly, systematic classroom observation can be a powerful tool in helping 
teachers improve their effectiveness by providing rich, in-depth data aboui specific classroom 
behaviors. 

Griffith (1973) identifies the following functions of systematic classroom observation: 

• To find out what learning activities students are engaged in and to appraise their value 

• To encourage and assist teachers to teach more effectively 

• To find out whether courses of study are related to students' needs and abilities and con- 
tribute to the goals of education 
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iMUts In use. Systematic classroom observation is a very powerful tool for documenting spe- 
cific teacher classroom behaviors. However, some important issues must be dealt with when con- 
sidering its implementation in a teacher evaluation system. Peterson and Kauchak (1982) note, 
"systematic observation provides a great deal of information about how well a teacher performs, 
but has significant limitations for evaluating overall teacher quality** (p. 37). They point out that 

observations can be disruptive to individual teaching patterns and that because perfor- 
mance judgments are context-dependent any comparisons among teachers or between 
observations of individual teachers are difficult and risky, (pp. 37-38) 

They further note that the use of systematic observation for formative teacher assessment is desir- 
able, but that the same is not true for summative evaluation. 

Although classroom observation is an important and desirable means of obtaining teacher eval- 
uative Information, the process presents problems related to validity and reliability. According to 
Glass (1974), unsuccessful attempts to rate teacher behavior in a reliable manner "stem largely 
from vague, general definitions of the behaviors to be rated and the lack of rater training" (p. 26). 
He recommends that no teacher characteristic be included in classroom observation rating ^«cales 
"until research has established both that it can be reliably observed and that it bears some signifi- 
cant relationship to desired pupil cognitive and affective states*' (pp. 27-28). 

Evertson and Holley (1981) also discuss the problem of reliability and validity related to sys- 
tematic classroom observation as a teacher evaluation strategy. They point out the difficulty of 
establishing any observation as a true representative sampie of student and teacher behaviors dur- 
ing a typical day. Observations are affected by the time of day, the subject being taught, the 
calendar of holidays and special events, and any number of other temporal factors (p. 94). 

As in classroom visits, rating error is an important issue in systematic observation. In the 
context of formative teacher evaluation, rating error may lead to identifying inappropriate 
improvement needs and result in wasted time and resources without any improvement in critical 
performance, in the context of summative evaluation, rating errors can result in grave consequen- 
ces and may have legal implications. 

Latham and Wexiey (1981) are among a growing body of experts who consider training raters 
in performance appraisal techniques absolutely necessary to reduce errors of judgment in evalua- 
tion. They point out, for example, that AT&T has paid compensation fees amounting to approxi- 
mately $12-15 million to employees who were victims of discrimination in promotions, transfers, 
and salary recommendations through rating errors. These two experts also identify six main types 
of rating errors, which can be summarized as follows: 

• The contrast effect error is the tendency for a rater to compare and evaluate a person rel- 
ative to ottier individuals rather than to rate him or her based upon the requirements of 
the job. 

• Flrtt*lmprestlon error or unfavorable judgments about an employee may cause the rater 
to ignore or perceptually distort subsequent information, so as to support initial 
impressions. 

• The halo effect refers to generalization from one aspect of a person's performance on the 
job to all aspects. 
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• The slmllar-to-me effect is the tendency for raters to reserve their highest ratings for per- 
sons whose backgrounds and attitudes are most similar to their own, even if these are not 
job related. 

• Ctntral ttnitenqf error is committed by the rater who plays it safe by consistently rating 
an employee on or close to the midpoint of a scale even when the employee's perfor- 
mance clearly warrants a substantially higher or lower rating. 

• Negative and positive leniency errors are committed by the rater who is either too hard or 
too easy in rating employees. In the performance appraisal process, positive leniency may 
raise unwarranted expectations of the employee for raises, promotions, or challenging job 
assignments. With negative leniency (or toughness) employees may get frustrated 
because no matter how hard they try, the rater cannot be satisfied. 

Rating errors also pose a significant problem in education and in classroom observations. 
Evertson and Holley (1981) document the fact that teacher organizations and unions regularly 
seek legal assistance regarding infringements arising from teacher observations or evaluations. 
They also strongly recommend the careful training of observers as an essential component of 
teacher evaluation. 

Instrumentation problems are an important issue related to all classroom observations. Coker 
(1985) has identified the following instrumentation problems affecting systematic classroom 
observation: 

• The extent to which the same observation instrument can be used in different grades and 
subject areas 

• The length and number of observations of teacher performance necessary to achieve reli- 
able assessment 

• The amount and kind of training an observer or rater must have to produce credible and 
accurate ratings of teachers 

• How often observers or raters need retraining in order to maintain their accuracy 

• The type of coding that provides the most valid and reliable data 

These critical issues need to be addressed so that more informed decisions can be made when 
using systematic observation as a teacher evaluation strategy. Coker also indicates the need to 
establish criteria for assessing the quality and usefulness of teacher evaluation instruments. 

Conditions foruee« A number of conditions must be met to implement systematic observation 
successfully in teacher evaluation. Peterson and Kauchak (1982) identify five essential characteris- 
tics of systematic observation, which are summarized as follows: 

• The observer is trained in the techniques of observation and is checked for actual reliabil- 
ity in practice. 

• The number and timing of visits are planned to ensure a fair and reliable sample of class- 
room time and events. (This may involve approximately eight sessions— depending on 
what is observed and how variable the activities in the classroom are.) 
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The focus of the observation is limited to a specific number of visible categories that have 
proved to be reliable and observable in practice. (Since there are limits to what trained 
observers can pay attention to, their attention needs to be focused.) 



• The recording system (checklists, entry forms, scoring) needs to be systematic, verifiable, 
and permanent. 

• Data should be analyzed with a single, coherent conceptual framework that has been sys- 
tematically validated to show its links with important features (e.g., student learning, 
school needs, legal expectations). 

Evertson and Holley (1981) suggest three essential characteristics of good teacher evaluations 
through classroom observation. These areas follows: 

• The individual to be observed is fully informed about the purpose and nature of the 
observation. 

• The entire process is conducted as unobtrusively as possible and with as little disruption 
to the normal routine as possible. 



Good communication between teacher and rater is maintained throughout the evaluation 
process, (p. 93). 



Pre- and postobservation conferences between the teacher being evaluated and the rater are 
necessary to capitalize on the full benefits of systematic observation. The preobservation confer- 
ence helps focus the evaluation efforts and set the evaluation guidelines. The postobservation con- 
ference provides for exchange, clarification* and counseling related to the evaluation and 
improvement strategies. 

Advantages and ditadvantages. The literature documents various advantages regarding the 
use of systematic observation as a teacher evaluation strategy. A summary follows of the most 
commonly cited points: 

• Observations can be focused on effective teaching practices that contribute to student 
achievement* such as— 

—clarity of presentation and explanation, 
—enthusiasm* 

—variety in the use of instructional materials and techniques, 

—task orientation and businesslike behavior, 

—provision for ample learning opportunities, 

—teacher use of student ideas* 

—use of multiple levels of discourse, and 

— -absence of negative criticism or probing. 

• Because of the ongoing nature of systematic classroom observation, improvement efforts 
can be closely monitored and corrective action taken to ensure successful achievement of 
improvements. 

• The frequent interaction between evaluator and teacher can help develop a collegial rela- 
tionship and an atmosphere of trust that alleviates the threatening aspects of evaluation. 
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• Frequent visits of evaluators in the classroom may help students get used to their pres- 
ence and act normally during observations. 

• Multiple visits enable evaluators to observe a broad variety of teacher behaviors under dif- 
ferent conditions, as well as allow them to observe a wide range of the teaching behaviors 
over time. 

• Systematic observations may result in more in-depth and richer data that can be gener- 
ated and translated into improvement reccr^mendations. 

• A trained observer, using an appropriate behavior-recording system, can generate valid, 
objective, and reliable teacher evaluation data. 

• Valid, reliable, and defensible observation records can be used to support personnel deci- 
sions, such as reassignment or termination. 

• Using videotaping techniques, individual teachers can rate themselves and reflect on their 
own teaching. Teachers can also make weekly videotapes, which can supplement direct 
classroom observations. 

Following are some of the major disadvantages of systematic classroom observations men- 
tioned in the literature: 

• Such observations are more suitable for formative evaluation than for summatlve 
assessment. 

• Classroom observations can be costly and time consuming. 

• Raters or evaluators must be adequately trained in systematic classroom observation, a 
process that requires time and money. 

• Instrument development is complex; consequently, commercial instruments may have to 
be purchased at additional cost. 

• The observation process can substantially alter teacher behaviors. 

• Observations of teacher behaviors may not cut across relevant subject-matter, grade- 
level, contextual, and situational factors. 

• The observation process may infringe on the privacy rights of students, and parents may 
oppose frequent evalua tions. 

Applications In vocational education. No evidence was found in the literature to refute or sup- 
port using systematic classroom observations in the assessment of vocational teachers. Addition- 
ally, little research exists about effective vocational teacher behaviors that contribute to students* 
learning. Most classroom observation research has been conducted at the elementary school level 
and in the general education area. Consequently, one cannot apply these research findings to 
vocational education settings with a high degree of confidence. However, with a sound guiding 
theoretical framework and proper instruments, systematic classroom and laboratory observation 
can be used to advantage in vocational teacher assessment. Figures 5-9 describe assessment 
instruments that are designed for use in conducting systematic classroom observations. 
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The time^n-task literature is an emerging area of research that has promising applications in 
vocational education. Time-on-task studies that make use of systematic observation can provide 
valuable information to vocational teachers for improving their teaching effectiveness (as well as 
potentially valuable instrumentation). The National Center for Research in Vocational Education 
has already conducted several time*on-task studies in vocational education settings at both the 
secondary and postsecondary levels. These studies indicate that postsecondary vocational educa* 
tion students spend more time on task than their counterparts in high schools. Because time on 
task is also related to student outcomes, it will surely be used more extensively in teacher evalua- 
tions in the future. However, the process still has limitations. For example, (1) students* on-task 
behaviors are often recorded without knowing the worth or the meaningfulness of the tasks being 
executed; (2) student interactions may be recorded as being off task, when in fact, considerable 
learning may be taking place in sharing ideas, (3) students engaged in cognitive tasks may be 
recorded as being off task; and (4) time-on-task research violat3S principles of individual differen- 
ces with respect to learning styles. In spite of these conceptual and methodological problems, 
time*on-task observations may become viable tools for use with other strategies in the assessment 
of vocational teachers. 
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TITLE: 


Classroom Observation Keyed for Effective Research (COKER) 


DEVELOPER: 


H.Coker and J.G.Coker 


DATE: 


1979 


AVAILABILITY: 


H.Coker and J.G.Coker 
P.O.Box 1017 
Mtnens, oeorgia ou 1 1 / 


Copyright: 


Yes 


Cost: 


Cost information not available 


DESCRIPTION: The COKER is an objectively administered, low-inference sign system for observing teacher/ 
student behaviors during classroon: sessions. 


Proctdurt: The COKER instrument consists of two sections, A and B. Section A involves reading a 5-minute 
observation regarding specific teacher and student transactions and/or interactions. Section 6 entails 
reordering observed teaching strategies and student and/or teacher cognitive and affective behaviors from 
memory following the previous 5-minute set of observation and recording of section A. Two 5*minute 
observations and recording periods using the COKER instrument are necessary during 1 observation and 
r xording periods using the COKER instrument are necessary during 1 observation session. 


ADMINISTRATION: 




When: 


During classroom observation sessions 


How: 


By observer through direct observation and systematic recordings 


Time: 


A total of 20*25 minutes to complete 10 minutes of observation and recording 


Scoring: 


The observation forms can be computer scored. Scoring keys and a user's manual 
are also available 


INSTRUMENT DATA: 




Reliability A^alldity: 


This instrument has been extensively used for research on classroom obiervation 
and has been found to be both valid and reliable. 



Figure 5. The Classroom Observation Keyed for Effective Research (COKER) 
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TITLE: 


Observation Schedule and Record, Form 5 (0SCAR5V) 


OEVEt.OPER: 


Donald M. Medley 


DATE: 


1979 


AVAILABILITY: 




For Furthtr 
Information: 


Donald M. Medley 
School of Education 
University of Virginia 
Charlottesville, Virginia 


Copyrights 


No 


DESCRIPTION: The 0SCAR5V is e verbal category system designed to describe the classroom learning environ- 
ment according to the relative frequencies of 80 different kinds of evenU in classroom Interaction (Medley, 
Coker,and Soar 1984). 


Numbtr of CttugoriM: The 0SCAR5V consists of 18 categories. Public Utterance-Nonsubstantive. Public 
Question-Substantive, Pupil Statement-Substantive, Pupil Response, Problem-Structuring Statement, 
Convergent Question, Elaboration 1 Question, Elaboration 2 Question, Divergent Question, No Evaluation, 
Considering-Supporting, Informing-Approving, Describing-Rejecting, Rebuking-Criticizing, Desisting. Pro- 
cedural, Nf utral-Nonsubstantive Question, and Positive. 


ADMINISTRATION: 




Whan: 


Dur-ng classroom instruction 


Time: 


Class duration 


Scoring: 


Forms are available that are scorable on DIGITEK and IBM computers. Computer 
programs for scoring and summarizing data generated by the 0SCAR5V are also 
available. 


RATER TRAINING; 


A minimum of 20 hours of training for raters in using the 0SCAR5V is necessary. 


INSTRUMENT DATA: 




Raliability/Validity: 


This instrument has evolved from 10 years of development efforts. Its validity and 
reliability has been weil established— reliability as high as .91 has been repoted. 


COMMENTS: 


For an in-depth discussion cf the OSCARS V, refer to Medley, Coker, and Soar 
(1984). 



Figure 6. The Observation Schedule and Record, Form 5 (OSCARSV) 
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TITLE: 



The Planning/Observation Instrument (P/OI) 



DEVELO?t:R: 
DATE: 

AVAILABILITY: 



Copyright: 
Cost: 



Teacher Assessment Unit 
Division of Staff Development 
Georgia Department of Education 

1985, revised version 

Teacher Assessment Unit 

Division of Staff Development 

Georgia Department of Education 

Atlanta, Georgia 30330 

or from the regional assessment centers (RAC) 

Yes 

Not indicated 



DESCRIPTION: The PA)1 designed to assess teacher competence on two dimensions: ( 1 ) planning for instruction, 
evaluating learner progress, and using acceptable written expression, and (2) observing teacher skill in the class- 
room. The planning component addresses the following competencies: 
1 . Plans instruction to achieve selected objectives 
2 Obtains information about the needs and progress of learners 
3. Demonstrates acceptable written and oral expression and knowledge of the subject 

The observation component focuses i ^ the following competencies: 

1 . Organize time, space, materials, and equipment for instruction 

2. Communicates with learners 

3. Demonstrates appropriate instructional methods 

4. Maintains a positive learning climate 

5. Maintains appropriate classroom t>ehaviors 

set of indicators for assessing these competencies is also provided. 

Intended Population: The P/01 is specifically designed for initial teacher certification purposes, but it can 
also be used in training, teacher preparation, and research. It is also promising for use in ongoing teacher 
evaluation. 



ADMINISTRATION: 
When: 

Time: 

Conditions for Use: 



Scoring: 
INSTRUMENT DATA: 
ReliabilHy/Validity: 



Twice per year, or until performance requirements have been reached on all 
competencies 

Entire class period 

Teachers must prepare and submit a portfolio representing a ^'-lO-day instructional 
unit for the assessment of the planning competencies. Teaching skills are evaluated 
through direct, scheduled observations of instruction as the teacher implements the 
lessons that have been planned and prepared in the portfolio. Teachers are rated by 
trained external evaluators and administrators. 

A user's manual provides guidelines for the scoring procedures. 



Both have been established in the instrument's developmental phase. 



Figure?. Planning/Observation Instrument (P/OI) 
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TITLE: 


Student Time-on-Task Observation Guides 
(Class Observation and Student Observation) 


DEVELOPER: 


Ida M. Halasz and Karen S. Behm 


DATE: 


1983 


AVAILABILITY: 


The National Center for Research in Vocational Education 
The Ohio State University 
itfou i\enny naoa 
Columbus, Ohio 43210-1090 


Copyright: 


No 


Cost: 


Not included 


DESCRIPTION: To determine how time is spent in vocational education classes 


Conttnt: The class observation guide is used to record the classroom activities of all the students and the 
teacher in a class during each minute. The student observation guide is used for observing 
individual students. 


Numbtr of Itmms: 


Not applicable 


Typt of Items: 


Coding observations 


Intended Populetion: 


Students and teachers 


Respondents: 


Observer 


ADMINISTRATION: 




When Adminitttred: 


Periodically 


How Administtred: 


On-site, in classroom 


Time Recommended: 


Class period 


Response Mode: 


Recording observations 


Scoring: 


Keypunched and machine scored 


TEST DATA: 




Reliability: 


Interrater reliability was found to be adequate. 


Validity: 


Content validity 



Figure 8. The Student Time-onTask Observation Guides 



61 

69 



TITLE: 


Flanders System of Interaction Analysis 


DEVELOPER: 


Ned A. Flanders 


DATE: 


1970 


AVAILABILITY: 


Refer to ^landers (1970) 

Addison— Welsey Publishing Company 


Copyright: 


Yes 


Cost: 


Not indicated 


DESCRIPTION: This observation instrument is used to analyze the influence of teachers in the classroom. 


Content: The instrument classifies teacher and student behaviors in a 10 x 10 matrix on the following 10 
categories: 




Teacher— Indirect Influences 

1. Accepts feeling 

2. Praises or encourages 

3. Accepts or uses ideas of students 

4. Asks questions 




Taacher— Direct Influences 

5. Lectures 

6. Gives directions 

1, Criticizes or justifies authority 




Student Categories 

8. Student talk-response 

9. Student talk-initiative 
10. Silence or confusion 


Number of Items: 


Not applicable 


Type of Items: 


Observation coding 


Intended Population: 


Teachers 


Respondent: 


Trained observer 


ADMINISTRATION: 
When: 


Periodically 


How: 


On-site, classroom 


Time: 


Class period 


Response Mode: 


Observation coding of behavior 


Scoring: 


Hand scored, frequency tabulation 


TEST DATA: 
Reliability: 


Coefficients of observer agreement have been reported to be as high as .85. 


Validity: 


Not indicated 


COMMENTS: 


For additional information, refer to Borich and Madden (1977). 



Figure 9. The Flanders System of Interaction Analysis 
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Studtnt Rating of Ttachtrs 



DMcriptlon 

This section discusses the use of student ratings in evaluating instructional staff. The discus- 
sion focuses on two basic r dthods of using student ratings: (1 ) student questionnaires and 
(2) student interviews. 

Of all the information sources available for use in evaluating instructional staff, student rating 
of teachers* performance is one of the most valid, reliable, defensible, readily accessible, efficient, 
and cost-effective. The use of student feedback for the «»valuation of instructors dates back to the 
Middle Ages. More formalized implementation started at Harvard and the University of Washington 
in the 1920s, when students began collecting the results of their assessment of instructors* perfor- 
mance to share with other prospective students (Eastridge 1976). 

The day-to-day close contact that students have with their teachers places students in an ideal 
position to document the teaching and learning transactions that take place in the classroom. 
Braskamp, Bradenburg. and Ory (1984) note that "as the primary recipients of instruction, students 
can provide an impartial and unique insight into the evaluation process** (p. 36). The student rat- 
ings approach consists essentially of asking students who have been exposed to the teaching of a 
particular instructor to document the teaching and learning transactions that went on in the class- 
room. These ratings are then used to evaluate the instructor*s effectiveness. 



Approaches and Techniques 

Student questionnaires. The student questionnaire is a paper-and-pencil instrument designed 
to elicit students* reactions to their teachers and their courses of study. Machine-scored question- 
naires are useful for collecting iarge amounts of data within relatively short periods of time. These 
data can be processed by a central system with speed and accuracy at relatively low cost. With van 
ious statistical software packages, the data can be subjected to different computer analyses to 
facilitate their interpretation and use. 

Machine-scored questionnaires use objective questions, by which students rate various 
aspects of teaching on a scale. Centra (1982) notes thaV the following dimensions of student rat- 
ings are commonly included in such questionnaires: 

• Organization, structure, or clarity 

• Teacher-student interaction or rapport 

• Teaching skill, communication, or lecturing abilit/ 

• Work load and course difficulty 

• Grading and examinations 

• Impact on students and student self-rated accomplithments 

• Global, overall ratings (p. 23) 
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Hand-scored questionnaires can be designed much the same way as machine-scoreo question 
naires; however, they can also use open-ended questions. Objective questions are easy to score 
using an answer key. Open-ended questions can generate rich, in-depth information but are much 
more difficult to score and assess. Decisions about which questionnaire format to use depend on 
what information is needed. Objective questions can cover a wide array of topics wilhin a short 
time. Open-ended questions are more time-consuming to administer and tend to be narrowly 
focused, but they yield more complete information and provide a better understanding of the top- 
ics u.ider investigation. Often, both formats are combined in a single instrument. 

Student interriewt. Student interviews can be used to elicit student reactions to a particular 
course or a particular teacher. This evaluation strategy is used less frequently because it is costly. 
Even so. it can yield important information not readily obtainable through student questionnaires. 
The interview setting offers a unique opportunity for interviewers to prompt for additional student 
feedback on responses requiring further clarification and insight. 

Sometimes it is useful to interview all or a sample of students in a particular course or of a par- 
ticular instructor. Student interviews can be structured or informal. Zemke and Kramlinger (1981) 
identify the following common features of a structured interview: 

First, the interview Is the only contact we will probably have with the informant. Second, 
the interviewee is in a unique position and is privy to information we can only obtain 
from him or her. Third, the interview is positioned as a formal, fact-finding affair; it is 
scheduled, planned, has rules of conduct, and a defined focus. Fourth, the results are 
formally analyzed in some fashion, (p. 100) 

Informal interviews tend to be more spontaneous than structured interviews. However, struc- 
tured interviews generally yield more objective, valid, and reliable information. The use of an inter- 
view protocol also helps to generate more objective data. 



Issuts In Use 

There is general consensus among educators and instructional developers that students play 
an important role in the revision of instructional materials during the instructional design process. 
The most common technique used to eMcit student input in this process is that of learner verifica- 
tion and revision (LVR). The rationale for evaluating and revising instructional materials is that if 
the assumptions made in the design process rest on weak learning theories, the resulting product 
may be weak or flawed. Systematic tryout of materials with students through LVR provides an 
opportunity to validate these assumptions empirically and to identify and correct errors in the 
materials. 

The use of student feedback to evaluate teaching raises substantial concern and controversy 
among educators. Some of these concerns are expressed by Aleamoni (1961) in the following 
passage: 

Faculty have solicited and chastized them (i.e., student ratings), whereas students and 
administrators have requested, used, misinterpreted, and misused them. Faculty, stu- 
dents, and administrators have all claimed, at one time or another, that the ratings are 
both reliable, valid, and useless, (p. 110) 

Student ratings of teachers can be used both formatively (for improvement of instructional 
delivery) or summatively (for personnel decisions). McGreal (1983) presents the following cau- 
tions, however, about using student feedback for summative purposes at the elementary and 
secondary school levels: 
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Teachers generally lack faith in the students* ability to accurately rate their performance, 
in many respects, their fears are justified. There is not a great deal of support for the 
accuracy of student ratings [at these levels], and the support that does exist is not 
enough to justify using student ratings in any summative evaluation sense, (p. 134) 

He adds the following: 

The major ingredient for successful use of student evaluation is the acceptance of the 
idea that students are much more reliable in describing life in the classroom than they 
are in muking evaluative judgement of the teacher, (p. 134) 

According to McGreal, student ratings should only be used to help beginning and nontenured 
teachers identify their weaknesses. Hs believes that because tenured, experienced teachers are at 
a "differeni stage of development, student evaluation should not be required of them** (p. 136) 
unless, of course, there is a joint agreement between the teacher and the supervisor. 

The use of student rating as a teacher evaluation strategy has not gained widespread use at 
the elementary and secondary school level, probably because of doubts and concerns about the 
age and maturity of the students and their ability to provide thoughtful and accurate information. A 
National survey conducted by the Educational Research Institute (Kowalski 1979) indicates that 
only a very small percentage of the elementary and secondary schools surveyed (1.7 percent of 
elementary schools, 1.8 percent of junior highs, and 2.1 percent of high schools) actually use stu- 
dents as a source of information for the evaluation of teachers. In most cases, the evaluation 
results are used for formative evaluation purposes. 

These cautions about using student feedback in evaluating elementary and secondary school- 
teachers is perhaps due in part to the paucity of research in this area. Peterson and Kauchak 
(1982) cite a few studies that support the use of student ratings at the elementary and secondary 
school level; on the other hand, Eastridge (1976) found disagreement among research findings on 
the validity of student evaluation at these levels. Eastridge does cite a few studies, however, that 
indicate that student feedback tends to be more effective than feedback from supervisors in chang- 
ing teacher behavior. ^ 

Another explanation for the apparent lack of interest in using student evaluation for assessing 
elementary and secondary schoolteachers may be related to the lack of resources necessary for 
processing, analyzing, summarizing, and interpreting the massive amount of information often 
collected from students. With the increasing availability of computers in schools and affordable 
software packages, however, these problems are vanishing. Special machine-scored student 
questionnaires can be designed for scoring using a microcomputer and an optical scanner. 

At the postsecondary level, student ratings of faculty effectiveness have become an accepted 
evaluation resource in many university and community college settings. Student evaluation cf 
postsecondary instructors has been legitimized by a substantial amount of research documenting 
its appropriateness, validity, and reliability. For example. Centra (1982) reports reliability coeffi- 
cients of .70 to .90 for student ratings of postsecondary teachers. Because reliability seems to be 
lower with a small sample of raters, however, evaluation results should be used only for improve- 
ment purposes. For personnel decision purposes, both the number of raters and the number of 
teachers and classes being rated should be given careful consideration in order to ensure high 
reliability. 

Research studies (e.g.. Centra 1982) have also shown that some factors have nothing to do 
with teacher effectiveness and cannot be controlled by the teacher, yet can directly influence stu- 
dent ratings. Some of these factors are class size, subject matter, type of course requirement (i.e., 
elective or required), and grade expected. 
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In the absence of further research on whether student ratings should remain anonymous, it is 
probably best to keep the process confidential. An effective, anonymous rating will avoid instru- 
ments that require writing, thereby eliminating any chance that a student's handwriting will be 
recognized. Unfortunately, this procedure limits the expression of student perceptions. Keeping 
student evaluations anonymous must also be considered when conducting face-io-face student 
interviews. In this case, using an interviewer such as a peer or another teacher from another 
school or different department may effectively preserve student anonymity. 



Advantages and Disadvantages 

Student rating is a viable source of teacher evaluation data. As with other techniques, how- 
ever, it has its strengths and weaknesses. The following are some of the advantages associated 
with student ratings: 

• Student ratings can generate valid, reliable, and defensible data if they are obtained 
carefully. 

• Students ire in a unique position to make valid judgments about their teachers and 
courses. 

• Other strategies only collect data from a slice in time, whereas student ratings can pro- 
vide a more complete picture of life in the classroom over longer periods of time. 

• Student ratings of teachers correlate with their achievements. 

• Teaching activities are often based upon weak assumptions and weak instructional 
theories; student ratings can validate teaching processes. 

• Comparisons of ratings across courses can provide unbiased estimates of overall teach- 
ing effectiveness. 

The major disadvantages of having students rate teacher performance are as follows: 

• If ratings are not kept anonymous, students may not reveal their true perceptions of their 
teachers* effectiveness. 

• If the process is not kept anonymous, it may have negative consequences for students. 

• Some extraneous factors, such as class size, subject matter, type of course requirements, 
and expected grade, may confound evaluation results. 

• Student ratings are perceived negatively by son^e instructors, more so at the elementary 
and secondary school level. 

• It is possible for teachers to manipulate student ratings in their favor. 

« Positive bias in student ratings can mislead teachers who need improvement. 

• Administrators would be advised to avoid using student ratings gathered for teacher 
improvement purposes when making personnel decisions. 



67 



ERIC 



75 



Appllcationi in Vocational Education 



The review of the literature and research on student ratings provides no special indicators to 
support or negate the use of this strategy in evaluating vocational teachers. In fact, no specific 
research studies related to using student ratings in vocational education were identified. The fol- 
lowing speculations about its applications, however, would seem useful: 

• Because of their long and continuous exposure to their teachers, vocational students are 
in a favorable position to evaluate their teachers* effectiveness reliably. 

• Vocational education students are, in general, mature enough to judge the quality of 
teaching to which they have been exposed. 

• Vocational students can evaluate their exposure to both classroom and laboratory 
teaching. 

• Vocational education is results oriented, which often leads to tangible project outcomes. 
This feature provides a unique and concrete means of comparing the effectiveness of the 
teaching processes to student outcomes. 

• Critical competencies required by vocational teachers have been empirically identified 
and validated nationally by the National Center. If these competencies are included in 
student rating forms, areas in which individual teachers need improvement can be accu- 
rately identified. The National Center has developed performance-based, individualized, 
self-study modules for all critical competencies needed by vocational teachers. These 
modules can be prescribed to address improvement needs. For example, if student eval- 
uations indicate that the course objectives were not clearly formulated, the National Cen- 
ter module entitled Develop Student Performance Objectives can be used to improve this 
set of skills. 

• Vocational education often includes a cooperative education component, whereby the 
school program is supplemented through training in business and industry. This gives 
vocational education students a unique opportunity to assess the relevance of their 
schooling to the occupational requirements and to their career objectives. 

• Because of the nature of vocational education, where instruction is frequently provided 
on a one-to-one basis, students have more opportunity to know their teachers and thus 
provide a more reliable assessment of teaching effectiveness. 

All of these features promise to make student rating a viable tool for the assessment of voca- 
tional teachers. Because these features tend to cut across the subject matter and levels of voca- 
tional education, they apply to both secondary and postsecondary teaching. 

The literature reflects very strong support for using student rating of instructional staff for 
formative evaluation purposes. Student rating of teachers can provide useful information not read- 
ily generated by other techniques. Further, in combination with other sources of information, stu- 
dent ratings ha. b the potential to improve vocational teacher quality and the quality of instruction 
In vocational education. Some of the instruments available for use in obtaining student input on 
teacher performance are described in figures 10-13. 
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TITLE: 


The Student Perceptions Instrument 


DEVELOPER: 


Teacher Assessment Unit 
Division of Staff Development 
Georgia Department of Education 


DATE: 


1985, revised edition 


AVAILABILITY: 


Teacher Assessment Unit 

Division of Staff Development 

Georgia Department of Education 

Atlanta, Georgia 30330 

or from the regional assessment centers (RAC) 


Copyright: 


Yes 


Cost: 


Not indicated 


DESCRIPTION: The Student Perceptions Instrument is intended to determine how students perceive the 

performance of their teacher in selected areas in order to identify improvement needs for staff develop- 
ment purposes. When using this instrument, the students rate their teachers on 35 competency indicators 
on a 3-point scale: NEVER, SOMETIMES, OFTEN. 


Intiindad Population: 


All learners from grade 3 and above 


ADMINISTRATION: 




Wh«n: 


During the middle of the school year or when students have had sufficient time to 
get to know their teachers 


Time: 


Not indicated 


Condition for Use: 


The instrument is administered by an external evaluator, in the absence of the teacher 
being assessed. Questions are read aloud to the students. 


Scoring: 


All completed instruments must be sent to the RAC for processing. 


INSTRUMENT DATA: 




Reliability: 


Not indicated 


Validity: 


Not indicated 



Figure 10. The Ctudent Perceptions instrument 
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TITLE: 


Student Perceptions of Teacher Style (SPOTS; 




DcVc LUPcR : 


Bruce W. Tuckman 




DATE: 


1970 




A\/AII ARII ITV* 






For Further 
Information: 


Rpfpr to Tuckman M970) 




Copyright 


Yes 




Cost: 


Not indicated 




DESCRIPTION: The instrument is designed to measure student perception of teacher directlveness. 




Conttnt: Each item consists of a stem followed by a response scale. Example: 






Your instructor is mainly concerned wlth- 




1 2 3 


4 5 6 7 8 


9 


The number of facts you know. Getting an Idea across to you. Your ability to think for yourself. 


Numbtr of Itmns: 


17 




Type of Items: 


Rating scale 




Intended Population: 


Teachers 




Respondent: 


Students 




ADMINISTRATION: 






When: 


Not indicated 




How: 


Self-administered 




Time: 


Not indicated 




Response Mode: 


Consumable instrument 




Scoring: 


Hand scored 




TEST DATA: 






Reliability: 


Interrater reliability was determined to be sufficient 




Validity: 


Not indicated 




COMMENTS: ^or additional information, refer to Borlch and Madden (1977). 




Figure 11. The Student Perceptions of Teacher Style Instrument (SPOTS) 
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TITLE: 

DEVELOPER: 

DATE: 

AVAILABILITY: 

For Further 
Information: 

Cost: 



Classroom Climate Questionnaire (CCQ) 

Herbert J. Watberg 

1966 

Refer toWalberg (1966) 
Not indicated 



DESCRIPTION: The CCQ is designed to assess the climate of the classroom as perceived by students. 
Contant: The CCQ yields 18 subscores on the following dimensions of classroom climate. 



1. Friction 

2. Classroom Intimacy 

3. Goal Direction 

4. Group Status 

5. Interest Heterogeneity 

6. Goal Diversity 



7. Group Status 

8. Democratic 

9. Subservient 

10. Satisfaction 

11. Strict Control 

12. Disorganization 



13. Alienation 

14. Personal Intimacy 

15. Stratification 

16. Egelitarian 

17. Formality 

18. Speech Constraint 



Number of Items: 80 
Type of Items: Likert type 

Intended Population: High school juniors and seniors 
Respondent: Students 
ADMINISTRATION: 

Periodically 
On*site, classroom 
Not indicated 

Paper and pencil; consumable instrument 
Hand or machine scored 



When: 

How: 
Time: 

Response Mode: 
Scoring: 
TEST DATA: 
Reliability: 
Validity: 



Split*half reliabilities for the 18 factors have been reported to range from .41 to .86. 
Validity of the CCQ is based on nonrandom standardization sample. 



COMMENTS: For additional information, refer to Borich and Madden (1977). 



Figure 12. The Classroom Climate Questionnaire (CCQ) 
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TITLE' 

III kC* 


An Instrument to Assess Student Attitudes toward Instruction 
(Student Instruction Attitude Inventory) 




DEVELOPER: 


Curtis R. Finch 




DATE: 


1970 




A\/AII ARII ITV* 
MVMIUMDILI IT* 






roi rurxnvi 
Informitton: 


The National Center for Research in Vocational Education 
The Ohio State University 
l9DU i^enny noaci 
'Columbus, Ohio 43210*1090 




Copyright: 


Yes 




Cost: 


Not Indicated 




DESCRIPTION: This instrument i^ designed to measure students' attitudes toward instruction. 




Contmt: Example- 


'The instruction was very boring. 




Strongly 
Disagree 


Disagree Neutral Agree 


Strongly 
Disagree 


Number of Ittmt: 


47 




Typt of Itiint: 


Likert type 




Inttndtd Population: 


Students in vocational education 




Rtspondtnt: 


Students 




ADMINISTRATION: 






Whtn: 


End of course evaluation 




How: 


On-site; classroom setting 




Time: 


10 minutes 




Rtsponia Modt: 


Paper and pencil; consumable questionnaire 




Scoring: 


Keypunched and machine scored, hand scored, optical scf/nning 




TEST DATA: 






Reliability: 


Not indicated 




Validity: 


Noc indicated 





Figure 13. Student instruction Attitude Inventory 
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The use of teacher evaluations of peers and colleagues is a concept that has been used exten- 
sively at the university and college level for some time. Such evaluations are gaining ground as an 
additional source of data for teacher evaluation at other levels of the educational system. The strat- 
egy discussed here has promise as an alternative for assessing vocational teachers. 



DMcrlption 

The peer review approach to teacher evaluation consists of a process whereby a colleague 
known to be competent both in a particular subject-matter area as well as in pedagogy documents 
the teaching practice of a fellow teacher through various evaluation strategies in order to establish 
the fellow teacher's instructional effectiveness. The peer review evaluation strategy is not in 
widespread use at the elementary and secondary school levels. Kowalski (1979) indicated that only 
3.0-3.5 percent of the sample elementary and secondary schoolo he surveyed reported using peer 
review as part of their teacher evaluation process. Growing public pressure for more and better 
teacher evaluations, however, has led some States and large public school systems to adopt the 
peer review process. 

Peer review can be used for making both staff improvement and personnel decisions. The sec- 
tions that follow describe the use of peer review for formative and summative teacher evaluation. 

Formative evaluation. When used as a staff improvement tool, the peer review technique can 
be implemented in two distinct forms: (1) teacher-mentor and (2) peer-to-peer reviews. In the 
teach^r-mwior mode/, the mentor is presumed to be a master teacher with broad knowledge, 
skills, and experiences. The mentor may assist the teacher comprehensively through such activi- 
ties as classroom observation, assistance in lesson planning, and assessment of student evaluation 
instruments. Although the responsibility for improvement of performance rests firmly on the 
shoulders of the teacher, the mentor acts as a role model, resource person, and linker to additional 
sources of assistance. Thus, the mentor must have good leadership skills and be capable of advis- 
ing adults. 

The one-on-one approach to evaluation may require considerable time and energy on the part 
of both teacher and mentor. The mentor needs time not only to evaluate and counsel but also to 
prepare before each intervention. The Charlotte-Mecklenburg (NC) program explicitly allocates 
one-half day a month for each provisionally certified teacher and mentor team to work together. 
The Toledo (OH) school system peer review plan assigns a full-time consulting teacher to no more 
than 10 new interns in any given school year (Lawrence 1985). 

In general, costs to the system may include the costs of replacing the mentor in the classroom, 
offering incentives to encourage teachers to take additional responsibilities as mr ntors. and pro- 
viding training for both mentors and interns. The benefits of such an approach, however, may well 
outweigh the costs involved. 

Investment in a formative peer review process early in the working life of a teacher helps 
establish sound professional practices— an invaluable foundation for building a successful teach- 
ing career. Problems are much more likely to be diagnosed and corrected early, resulting in 
greater teacher satisfaction and instructional quality. This is especially relevant in vocational edu- 
cation, where many new teachers have little or no previous training in pedagogy or experier ce in 
teaching. 
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in addition to the potential benefits to the intern teacher and the students, the mentor process 
also oflers advantages for the mentor. Mentors may use the process to develop or refine their own 
leadership, counseling, and teaching skills, increased monetary gains, higher status, and renewed 
motivation repressnt additional rewards to mentors. Teachers may benefit by developing a forum 
for discussion of their craft among others, rather than experience the frustration and isolation that 
has often characterized the profession in the past. 

The petr-to*pter model is also used in the context of formative evaluation, that is, staff 
improvement. The major characteristics that differentiate the peer-mentor Bn6 the pe^r-to*peer 
processes are (1) the relationship of equal status existing between peers, as opposed to the super- 
ior status of the mentor in the peer-mentor relationship; and (2) the level of formality of each pro- 
cess. Peer-to-peer review may be more informal, with no specified time frame, or it may revolve 
around a specific staff development activity. 

Stallings (1984) has devised a model whereby peers observe each other in the classroom and 
''collect data" to be used by the observed teacher only fcr improvement purposes. Alt information 
collected is strictly confidential, and participating teachers sign an oath to this effect. The types of 
information to be collected, as suggested by Stallings, include the specific behaviors of students 
and teachers in a framework that allows for ttme-on-task analysis. (This, of course, may be varied 
according to the intended focus of the ctaff development.) 

Another option has supervisor, administrator, or other facilitator guide a group of six to eight 
teachers in interpreting and analyzing the results of their mutual observations. Another approach 
involves viewing and analyzing videotapes of a faculty member by peers as a growth expp^* ice 
for the entire group. 

One form of peer-to-peer review can occur in the process of team teaching. This particular 
model has been useo successfully in a vocational setting in which a vocational instructor is paired 
with a technical expert from industry to teach a specific topic in which the teacher has only a 
liPTiited knowledge. The vocational instructor provides the instructional expertise and the technical 
expert brings the technical knowledge. The two may work together to develop a curriculum that 
facilitates students* learning while providing technically up-to-date information. Each peer may 
learn something from the other in the process. 

Similarly, pairs or small groups of instructors with various levels o^ subject-matter expertise 
and interest may use the team teaching approach to observe each other's teaching and to improve 
their cwn instructional strategies. Each of these peer review' techniques is geared toward 
improvement and each necessitates the open, honest sharing of information among peers. Diag- 
nosing problems and evaluating teacher performance is the joint responsibility of the reviewer and 
reviewee and, therefore, impiibs a truct on the part of each that the information so obtained and 
shared wiSi not be used to harm the reviewee in any way. 

Summative eviiiuation. The tradition of peer review in summative evaluation is the foundation 
of the faculty personnel decision-making process at the university and college level. In this con- 
text, peer review serves any combination of the following purposes:faculty appointments, promo- 
tions, granting of tenure, selection of manuscripts for publication, and approval of research grants. 
Since the introdu'^tion of summative peer review at the elementary and high school level is very 
recent, most of the issues, arguments, and methods discussed here relate to application at the 
pos^secondary level. 
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One of the major arguments in favor of using peers in the 3ummative evaluation process is that 
since peers have had experiences involving the conditions in a given school and in similar or iden- 
tical classrooms, they will be more likely to make valid judgments about the appropriateness of 
various teaching practices and behaviors. The problem with this approach, of course, lies in the 
lack of certainty with which such Judgments can be made and the lack of knowledge related to 
actual student outcomes. This point is illustrated by Wise et al. (1984) in the following case-study 
example: 

Through interviews with teachers and adminisirators who had served on remediation 
teams, the authors noticed a systematic intolerance of teachers who departed too bla- 
tantly from conventional strategies, even if they were educationally effective, (p. 9) 

Clearly, peers bring their biases, values, and understandings of the world with them to the eval- 
uation process. Thus, the use of peers in summative evaluation involves questions as to the validity 
and reliability of their judgments. For example. Centra (1982) found that rating of teacher class- 
room performance by peers and administrators is often based primarily on teachers* reputations 
and evidence from student evaluations. When peers rate colleagues* overall instructional effective- 
ness on the basis of actual classroom observation (with no prior exposure to student evaluations 
or information about teacher **reputation*'), the ratings tend to be overinflated and unreliable, and 
the results do not correlate with student ratings or ratings performed by administrators. 

Many studies conclude that the notion that peers are somehow inherently better prepared and 
in a better position to evaluate the teaching effectiveness of their collf^agues is a misconception. 
Clearly, several factors will make a difference in their ratings. Included among these are the 
following: 

• The ability of peers to assess performance, which depends upon the number of repeated 
observations of random and representative samples of performance 

• Training of observers with regard to the observations expected and observational and 
recording techniques 

• The availability and quality of additional "evidence /* such as student evaluations 

• The nature of the rating instrument and whether it elicits responses on specified behav- 
iors or on overall performances 

• Whether multiple sources of independent observations are available 

In order for peer evaluation to be successful, Dienst (1981) suggests the following guidelines: 

• A sufficiently large number of colleague evaluators should be selected. 

• Colleagues should have fair knowledge of the subject matter, and they should be per- 
ceived as being fair. 

• Each colleague should make at least two visits. The faculty being evaluated should partic- 
ipate in organizing the classrcn i visits to ensure that the nature of the particular classes 
visited are more or less typical, especially where a variety of teaching methods is used 
(i.e., iecture, discussion, student presentations, labs, and shops). 
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• The criteria by which colleagues are to make observations and evaluations should be 
made explicit and should reflect broad departmental consensus. 

• Information collected by peers should reflect only what they are in a position to observe 
(i.e.. coverage of a topic or appropriateness of teaching strategy and student participa- 
tion, as opposed to availability outside of class and student participation). 

• Confidentiality of sources and findings is required if the peer review process is used for 
personnel decisions. This is less necessary when the information is used for formative 
purposes only. 



IMUM in Use 

Because peer evaluation, more than any of the other strategies examined, implies a shift in the 
roles and responsibilities of both teachers and administrators, what happens during the transition 
stage is critical to the success or failure of this type of program. No transition is entirely smooth or 
without its opponents; however, in anticipation of potential problems, the following suggestions 
are offered: 

• Include both teachers and administrators at all levels if they will be affected by the out- 
comes in the decision-making process. 

• Provide ongoing training for both teachers and administrators in both the evaluation pro- 
cesses and in mentoring and leadership roles. 

• Encourage frank, open communications with a sincere effort to try to solve peer review 
problems as they arise 

Of the school systems visited in the conduct of this study, two used peer reviews as an integral 
part of the overall evaluation process. The Toledo (OH) public school system used 20 consulting 
teachers, selected on the basis of their exemplary teaching to assist new intern teachers as well as 
veteran teachers who might be having specific difficulties. Once in place, the peer review system at 
Charlotte-Mecklenburg (NO) will draw from a pool of "Career i" (highly experienced) teachers to 
act as mentors for new and provisionally certified teachers and to serve as members of their provi- 
sional advisory assessment (PAA) teams. 

In both of these school systems, the peer reviewers are experienced and respected teachers 
who have additional training in assessment and mentoring techniques. Although the primary role 
of these individuals ts one of formative evaluation, they also participate in the summative decision- 
making process as It affects new. provisionally certified teachers. In the Toledo school system, 
where peers are also uS€<J to help diagnose and intervene in cases of tenured teachers, their partic- 
ipation is purely formative !n nature ('■'oledo Public Schools 1983). 

The extent to which development and growth occurs as a result of the peer process wilt 
determine its usefulness. This is an iroportant consideration for administrators and teachers alike, 
because there are many ways in which evaluation and staff development funds and other adminis- 
trative resources might be allocatet. One test of the usefulness of formative peer reviews as an 
alternative for teacher evaluation is whether the results of staff development that were planned on 
the basis of peer reviews are more effective, less effective, or comparable tc those achieved 
through alternate programs or approaches. 
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Clearly, peer selection and matching can have considerable impact on the effectiveness of the 
overall process. Selection of mentors requires a much more extensive screening process than 
selection of individuals to participate in a peer-to-peer review process. Knowledge of facilitation 
skills, a keen understanding of human development, and appreciation for the needs of adult 
learners are skills that should be required of peer and mentor reviewers if the peer review process 
is to succeed. 



AitvanUiges and DisadvanUiges 

Although peer review is a viable option in teacher evaluation, it is not a panacea. As with other 
aporoaches, peer review has its strengths and weaknesses. The following advantages of peer 
review are associated with the implementation of peer review in a teacher evaluation system: 



• Peer review helps make teachers more responsible professionals capable of monitoring 
their own growth and development as well as those of their colleagues. 

• If implerrented properly and within the appropriate organizational climate, peer review 
can remove the threatening aspects of performance appraisal. 

• Peer review can broaden the focus of teacher evaluations to include the content area. 
Traditionally, evaluation activities have been restricted to teaching effectiveness. Peer 
review brings in subject-matter expertise as an additional dimension to the evaluation 
process. 

« The collegial approach to evaluation through peer review can help change the negative 
connotation of teacher evaluation. 



• Peer review provides opportunities for more frequent evaluation and more direct monitor- 
ing of interventions than do other methods. 

• Peer review can help mitiimize problems associated with the validity end reliability of eval- 
uation data. 



The following are some disadvantages associated with using peer review to evaluate the per- 
formance of vocational education teachers: 



• Peer review can be costly to implement. 

• The appropriate organizational climate necessary for successful peer review implementa- 
tion may be difficult to create. 

• Training and support for peer reviewers and mentors are critical for successful implemen- 
tation. When peer review is used in systems hav'ng other summative evaluation mecha- 
nisms in place, conflicting information from all evaluation techniques can create serious 
problems and lack of confidence in the results. 

• In schools where many teachers have little real commitment to teaching, the process will 
result in little improvement. 
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Applications In Vocational Education 



The peer review process has great potential for application in vocational teacher assessment 
fiimed at tpacher improvement, although its application for personnel decisioiiS is not clear. The 
most important condition for successful implementation of the peer review proces;* in vocatirnal 
education is the existence of an appropriate organizational climate. A supportive and collaborative 
organizational climate, open communication channels, and an administration that supports and 
festers academic excellence are required. In addition, peer reviews probably should not be used in 
a school system experiencing staff surpluses that are the result of decreasing enrollment because 
such a practice can generate an undesirable climate in the school. 

Implementation of peer review, however, should not create a sense of competition within the 
school. Improvement should be viewed as a team effort, and all team members must feel that they 
have the entire support of the school. Teachers should be involved in the choice of their peer 
reviewers, and personality conflicts that may arise between peers must *^ identified and resoived. 

In vocational education, peer reviewers need not always be matched on the basis of subject- 
matter expertise. In fact, there can be beneficial matches between vocational teachers and aca- 
demic teachers. Mutual collabo'^ation between them can help make the teaching of basic skills 
more effective and relevant to vocational education students and enhance their infusion into voca- 
tional education teaching. 

Adequate training should be provided for both school administrators and teachers before the 
peer review process t>egins. The confidentiality of the information gathered through peer reviews 
must be guaranteed. Also, prior agreement is necessary to ensure that information gathered for 
staff improvement purposes will not be misused for personnel decisions. 

A qualified, neutral arbitrator should be identified and available to settle conflicts between 
peers and to provide necessary teacher counseling that may be beyond the capabilities of peer 
reviewers. T!me for peer counseling, as well as adequate time for preliminary preparation for such 
activities, should form part of the reviewers' and mentors* normal work loads. Salary incentives are 
appropriate for these additional responsibilities. 

The conditions related to the successful impiementation of the peer review process in voca- 
tional education that have been outlined in this section by no means constitute an exhaustive list. 
Rather, they reflect some of the major concerns that must be considered in implementing a peer 
review process. 
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Sflf-Evaluatlon 



DMCription 

The teacher self-evaluation strategy is based on the premise that, as presumably competent 
professionals, teachers ought to be able to critique their own performance. The validity of this 
premise may vary, however, depending upon who requests the information, the purpose of the eval- 
uation, and the approach or method needed to obtain the information necessary to make the 
judgment. 

The purpose of teacher self-evaluation may be as follows: 

• Informatlva— Teachers provide descriptive information about their activities and accom- 
plishments that may be highly relevant for assessing performance but difficult for others 
to obtain by other means. 

• Evaluathra/Judgmantal— Teachers rate their performance on rating scales, question* 
naires, or in other forms of self-reports, such as reacting to videotapes. Factual reports/ 
documents provided by the teacher may be used as a part of an overall summative 
decision-making process. This is a particularly common practice at the postsecondary 
level, where faculty are expected to submit curriculum vitae of their accomplishments as 
well as other documentation of their performance (e.g., papers, and so forth) to be evalu- 
ated by their peers and administration for tenure and other personnel decisions. On the 
other hand, it is not normally deemed proper for teachers to be asked to make evaluative 
or judgmental statements of their own performance (or, for that matter, even 
developmental-type statements) for the purpose(8) of summative evaluations. 

• Dtvelopmental— Teachers express an interest, desire, or need for further development in 
specific areas and via specific learning mechanisms. Teacher self-assessment may be 
used to determine development needs not only of individual teachers, but also of groups 
of teachers, and to prioritize overall staff development programs so they wiil do the most 
good. 

Teacher self-assessment may involve a variety of approaches or techniques. Iwanicki and 
McEachern (1&84) offer a set of strategies for teacher self -assessment that are based upon the 
type of information. These are as follows: 

• Individual Aatessments— These may include personal reflection, an analysis of classroom 
audiotapes or videotapes, and self -assessment checklists They always involve the 
teacher in the assessment process. They provide information about teacher behavior that 
is known to the teacher and that he or she may or may not choose to share with others. 

• Feedback Asaessments— These may include student evaluation ratings provided by peer 
teachers and supervisory staff, and Interactive assessments, such as microteaching video- 
tapes. These strategies involve others in the assessment process, at the request of the 
teacher. Interactive assessment may, in addition, provide a forum for the teacher and oth- 
ers to discover new information about teacher behavior that was unknown prior to 
assessment. 

Most theoretical !iterature on self -evaluation focuses upon individual assessments, whereas 
other literature reveals that most self-evaluation programs possess a more comprehensive nature 



81 



ERLC 



89 



that involves some feedback and interaction. Recognizing that instructors may use multiple evalua- 
tion techniques in deriving information for self*evaluation, Carroll (1981) discusses five techniques 
or aids that are unique to self-evaluation: self-rating forms, self-reports, self-study materials, 
observation of colleagues' teaching, and videotape and audiotape feedback of one's own teaching. 

Centra (1982) reports that the use of self-reports and evaluations by university department 
chairpersons for purposes of promotion and tenure decisions is minimal. Even where this informa- 
tion is used for summative evaluation, department chairpersons surveyed by Centra ranked self- 
evaluation as ninth among a set of IS criteria. 

No comprehensive data currently exist on actual use of self-evaluations for formative or sum- 
mative purposes, but there is ample testimony to the importance and relevance of teacher self- 
evaluation for the purposes of staff development and improvement. Self-evaluations are used by 
teachers at ali school levels, and the process is as suitable for vocational instructors as it is for 
academic instructors. 



Issues In Use 

Studies (e.g.. Centra 1973; Webb and Nolan 1955) of the relationship between student ratings 
of teacher performance and teacher self-ratings have found correlations from as high as .62 to as 
low as .21. Centra also found that teachers with no previous exposure to student ratings tend to 
rate their performance higher than do their students. However, once given feedback from results of 
student ratings, these teachers tend to decrease their self-ratings (Braskamp and Caulley 1978) or 
they improved their teaching (Centra 1972), so that subsequent comparisons show a higher corre- 
lation between student ratings and self-ratings. 

Blackburn and Clark (1975) found considerable divergence between teacher self-rating and 
ratings by colleagues and administrators. On the other hand, they found correlations of the latter 
two to be much higher with each other as well as with student ratings. At the high school level, 
McAfee (1975) found no significant correlation between the responses of teachers and supervisors 
on estimates of teacher abilities, professional skills, attitudes, public relations, and knowledge and 
use of various methods and techniques of teaching. 

All of these data, however, are based upon assessments of absolute levels of performance. 
Studies that compare relative strengths and weaknesses of teacher performance in specific sets of 
skills show much higher correspondence (.77) between student and self-ratings (Centra 1973; 
Marsh. Overall, and Kesler 1978). 



Advantagts and Disadvantages 

Carroll (1981) provides an excellent description of many of the advantages and disadvantages 
of teacher self-evaiuations, as well as some conditions to be considered for their appropriate use. 
The following arguments are abstracted. 

Advantages. If the ultimate goal of the evaluation process is teacher development or improve- 
ment, then this form of evaluation is useful in that individual teachers are more likely to act upon 
self-gained data than upon information from other; sources (Centra 1972). Also, instruments, pro- 
grams, and projects that are based upon input/feedback from self-evaluations are more likely to be 
perceived as relevant by the teachers for whom they are developed. 
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DiMdvanUig«t. The disadvantages of teacher self-evaluation deal primarily with the type of 
self-evaluation that is based solely upon individual assessment, rather than self-evaluation that 
incorporates feedback or interaction with others. The majority of self-evaluation data lack credibil 
ity for most decisions except those involving self-development. The primary concern is the ability 
of the individual teacher to assess his or her own performance accurately and objectively. This 
concern is particularly acute when the purpose of the evaluation is to make summative types of 
decisions, although many question its validity even in less threatening situations as well. 



Applications In Vocational Education 

Thus far. no reports have been found that specifically outline the roles of teachers* self- 
evaluation in vocational education. However, this is no indication that the technique as used m 
traditional academic teaching settings cannot also be adapted to vocational teacher evaluation. 
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